Average and standard deviation over various time intervals. Simple

T

tobesus

Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
K

Ken Sheridan

I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England
 
T

tobesus

Thanks Ken. I changed my field names to SampleDate and SampleTime and ran the
query you recommended and it came very close to working.

The query parsed the data into five second intervals as intended, but didn't
actually average the data over five seconds. Instead it averaged the data
over one data point (every fifth second) and therefore produced no standard
deviation values because there was no deviation.

I noticed in my original table that my SampleDate and SampleTime fields were
in Date/Time format, not Text format as you were assuming, and I'm wondering
if that's the problem.

I tried changing them to Text format and tried to save; Access would work on
this for a few seconds and then report that "Access cannot change the data
type. There isn't enough disk space or memory."

I'm working with 1.5 million rows of data (x 3 fields columns) and am using
a 2-yr old Toshiba laptop with 1.23 GB of DDR SDRAM.

Any thoughts? Thanks again for your help!

Ken Sheridan said:
I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England

tobesus said:
Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
K

Ken Sheridan

Mea culpa, I wasn't thinking straight. The query shouldn't return the
individual date/time values at all, just restrict the set over which the
average and standard deviation functions operate to those values, i.e.

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0;

Its not the format of the values which is important, but the data type of
the columns in the table. Are you sure they are of date/time data type. If
they are then the query would I think still be restricted to the correct rows
using the expression I gave you, but that would be fortuitous because of the
default format in which the value is shown (its actually a 64 bit floating
point number under the skin). If they really are of date/time data type a
better way would be to use the Second function:

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([Time]) MOD 5 = 0;

Ken Sheridan
Stafford, England

tobesus said:
Thanks Ken. I changed my field names to SampleDate and SampleTime and ran the
query you recommended and it came very close to working.

The query parsed the data into five second intervals as intended, but didn't
actually average the data over five seconds. Instead it averaged the data
over one data point (every fifth second) and therefore produced no standard
deviation values because there was no deviation.

I noticed in my original table that my SampleDate and SampleTime fields were
in Date/Time format, not Text format as you were assuming, and I'm wondering
if that's the problem.

I tried changing them to Text format and tried to save; Access would work on
this for a few seconds and then report that "Access cannot change the data
type. There isn't enough disk space or memory."

I'm working with 1.5 million rows of data (x 3 fields columns) and am using
a 2-yr old Toshiba laptop with 1.23 GB of DDR SDRAM.

Any thoughts? Thanks again for your help!

Ken Sheridan said:
I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England

tobesus said:
Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
T

tobesus

Thanks again Ken. Unfortunately with this new query I'm only getting back one
single row: the average and st dev of ALL the data apparently.

When I look at my table in design view, my SampleDate and SampleTime fields
have a data type listing of Date/Time, but both of the queries you suggested
produced the same result.

Any idea what the problem could be? Thanks again!

Ken Sheridan said:
Mea culpa, I wasn't thinking straight. The query shouldn't return the
individual date/time values at all, just restrict the set over which the
average and standard deviation functions operate to those values, i.e.

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0;

Its not the format of the values which is important, but the data type of
the columns in the table. Are you sure they are of date/time data type. If
they are then the query would I think still be restricted to the correct rows
using the expression I gave you, but that would be fortuitous because of the
default format in which the value is shown (its actually a 64 bit floating
point number under the skin). If they really are of date/time data type a
better way would be to use the Second function:

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([Time]) MOD 5 = 0;

Ken Sheridan
Stafford, England

tobesus said:
Thanks Ken. I changed my field names to SampleDate and SampleTime and ran the
query you recommended and it came very close to working.

The query parsed the data into five second intervals as intended, but didn't
actually average the data over five seconds. Instead it averaged the data
over one data point (every fifth second) and therefore produced no standard
deviation values because there was no deviation.

I noticed in my original table that my SampleDate and SampleTime fields were
in Date/Time format, not Text format as you were assuming, and I'm wondering
if that's the problem.

I tried changing them to Text format and tried to save; Access would work on
this for a few seconds and then report that "Access cannot change the data
type. There isn't enough disk space or memory."

I'm working with 1.5 million rows of data (x 3 fields columns) and am using
a 2-yr old Toshiba laptop with 1.23 GB of DDR SDRAM.

Any thoughts? Thanks again for your help!

Ken Sheridan said:
I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England

:


Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
K

Ken Sheridan

What do you actually want? You can restrict the query over a date range,
which will return a single row:

PARAMETERS
[Enter start date:] DATETIME,
[Enter end date:] DATETIME;
SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([SampleTime]) MOD 5 = 0
AND [SampleDate] BETWEEN
[Enter start date:] AND [Enter end date:];

Or you can group it by a time period, e.g. month, which will return one row
per month:

PARAMETERS
[Enter year:] SHORT,
[Enter month:] SHORT;
SELECT
YEAR([SampleDate]) AS [Sample Year],
MONTH([SampleDate] AS [Sample Month],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([SampleTime]) MOD 5 = 0
AND YEAR([SampleDate]) = [Enter year:]
AND MONTH([SampleDate]) = [Enter month:]
GROUP BY YEAR([SampleDate]), MONTH([SampleDate]);

You could of course combine the two, restricting the query by a date range
and grouping within that range.

Both of the above queries use parameters which will prompt you to enter the
values when you run the queries. I've assumed the columns are of date/time
data type. If they are text data type then to return the year and month the
corresponding string expressions to YEAR([SampleDate]) and
MONTH([SampleDate]) would be LEFT([SampleDate],4] and MID(SampleDate],6,2)
and the parameters for these would declared as TEXT(4) and TEXT(2) rather
than as SHORT. In the first query the parameters would, if the column is
text, be declared as TEXT(10) and the values entered in exactly the same
format as the data in the table (be careful here that the actual values are
not in fact like 20061002 and what you are seeing is a formatted value).

Ken Sheridan
Stafford, England

tobesus said:
Thanks again Ken. Unfortunately with this new query I'm only getting back one
single row: the average and st dev of ALL the data apparently.

When I look at my table in design view, my SampleDate and SampleTime fields
have a data type listing of Date/Time, but both of the queries you suggested
produced the same result.

Any idea what the problem could be? Thanks again!

Ken Sheridan said:
Mea culpa, I wasn't thinking straight. The query shouldn't return the
individual date/time values at all, just restrict the set over which the
average and standard deviation functions operate to those values, i.e.

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0;

Its not the format of the values which is important, but the data type of
the columns in the table. Are you sure they are of date/time data type. If
they are then the query would I think still be restricted to the correct rows
using the expression I gave you, but that would be fortuitous because of the
default format in which the value is shown (its actually a 64 bit floating
point number under the skin). If they really are of date/time data type a
better way would be to use the Second function:

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([Time]) MOD 5 = 0;

Ken Sheridan
Stafford, England

tobesus said:
Thanks Ken. I changed my field names to SampleDate and SampleTime and ran the
query you recommended and it came very close to working.

The query parsed the data into five second intervals as intended, but didn't
actually average the data over five seconds. Instead it averaged the data
over one data point (every fifth second) and therefore produced no standard
deviation values because there was no deviation.

I noticed in my original table that my SampleDate and SampleTime fields were
in Date/Time format, not Text format as you were assuming, and I'm wondering
if that's the problem.

I tried changing them to Text format and tried to save; Access would work on
this for a few seconds and then report that "Access cannot change the data
type. There isn't enough disk space or memory."

I'm working with 1.5 million rows of data (x 3 fields columns) and am using
a 2-yr old Toshiba laptop with 1.23 GB of DDR SDRAM.

Any thoughts? Thanks again for your help!

:

I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England

:


Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
T

tobesus

Sorry Ken, let me try and do a better job of explaining what I want. Right
now I have one data point per second, and what I want to end up with is one
data point per 5 seconds - which would be the average (and stDev) of the last
5 seconds of data.

So if my table contained ten seconds of data like this:
SampleDate SampleTime XData
12/19/2006 21:36:01 0.028
12/19/2006 21:36:02 0.026
12/19/2006 21:36:03 0.026
12/19/2006 21:36:04 0.021
12/19/2006 21:36:05 0.026
12/19/2006 21:36:06 0.026
12/19/2006 21:36:07 0.026
12/19/2006 21:36:08 0.024
12/19/2006 21:36:09 0.024
12/19/2006 21:36:10 0.024

Then I would like my query to return two rows of data (one every 5 seconds)
like this:

SampleDate SampleTime Avg StDev
12/19/2006 21:36:05 0.0254 0.002608
12/19/2006 21:36:10 0.0248 0.001095

-where the average and standard deviation for the first five seconds are
represented by the first row of the query, and the average and standard
deviation for the next five seconds are represented by the second row, and so
on.

I would also like to be able to average over other intervals like 3s or 10s,
etc., which I'd imagine is just a matter of number subsitution in the SQL
once we get that worked out.

So if I have 100 seconds of table data, averaging at 5s intervals, I should
end up with 20 rows returned in my query. The queries I've tried so far have
all returned just one row of data per query.

Thanks again for your help and patience!


Ken Sheridan said:
What do you actually want? You can restrict the query over a date range,
which will return a single row:

PARAMETERS
[Enter start date:] DATETIME,
[Enter end date:] DATETIME;
SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([SampleTime]) MOD 5 = 0
AND [SampleDate] BETWEEN
[Enter start date:] AND [Enter end date:];

Or you can group it by a time period, e.g. month, which will return one row
per month:

PARAMETERS
[Enter year:] SHORT,
[Enter month:] SHORT;
SELECT
YEAR([SampleDate]) AS [Sample Year],
MONTH([SampleDate] AS [Sample Month],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([SampleTime]) MOD 5 = 0
AND YEAR([SampleDate]) = [Enter year:]
AND MONTH([SampleDate]) = [Enter month:]
GROUP BY YEAR([SampleDate]), MONTH([SampleDate]);

You could of course combine the two, restricting the query by a date range
and grouping within that range.

Both of the above queries use parameters which will prompt you to enter the
values when you run the queries. I've assumed the columns are of date/time
data type. If they are text data type then to return the year and month the
corresponding string expressions to YEAR([SampleDate]) and
MONTH([SampleDate]) would be LEFT([SampleDate],4] and MID(SampleDate],6,2)
and the parameters for these would declared as TEXT(4) and TEXT(2) rather
than as SHORT. In the first query the parameters would, if the column is
text, be declared as TEXT(10) and the values entered in exactly the same
format as the data in the table (be careful here that the actual values are
not in fact like 20061002 and what you are seeing is a formatted value).

Ken Sheridan
Stafford, England

tobesus said:
Thanks again Ken. Unfortunately with this new query I'm only getting back one
single row: the average and st dev of ALL the data apparently.

When I look at my table in design view, my SampleDate and SampleTime fields
have a data type listing of Date/Time, but both of the queries you suggested
produced the same result.

Any idea what the problem could be? Thanks again!

Ken Sheridan said:
Mea culpa, I wasn't thinking straight. The query shouldn't return the
individual date/time values at all, just restrict the set over which the
average and standard deviation functions operate to those values, i.e.

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0;

Its not the format of the values which is important, but the data type of
the columns in the table. Are you sure they are of date/time data type. If
they are then the query would I think still be restricted to the correct rows
using the expression I gave you, but that would be fortuitous because of the
default format in which the value is shown (its actually a 64 bit floating
point number under the skin). If they really are of date/time data type a
better way would be to use the Second function:

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([Time]) MOD 5 = 0;

Ken Sheridan
Stafford, England

:

Thanks Ken. I changed my field names to SampleDate and SampleTime and ran the
query you recommended and it came very close to working.

The query parsed the data into five second intervals as intended, but didn't
actually average the data over five seconds. Instead it averaged the data
over one data point (every fifth second) and therefore produced no standard
deviation values because there was no deviation.

I noticed in my original table that my SampleDate and SampleTime fields were
in Date/Time format, not Text format as you were assuming, and I'm wondering
if that's the problem.

I tried changing them to Text format and tried to save; Access would work on
this for a few seconds and then report that "Access cannot change the data
type. There isn't enough disk space or memory."

I'm working with 1.5 million rows of data (x 3 fields columns) and am using
a 2-yr old Toshiba laptop with 1.23 GB of DDR SDRAM.

Any thoughts? Thanks again for your help!

:

I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England

:


Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
K

Ken Sheridan

What's needed is an expression by which the rows for each n second range can
be grouped, so that the aggregation operators can then be applied to the
group. So the result set shows the last value of the group we need a
function which will return that value for any of the n values. By simply
adding the SampleDate and SampleTime values (assuming they are true date/time
values) we get a unique single date time value per row, so that's the
starting point. I think it will be easier to write a little VBA function to
return the required value for each group rather than trying to do it with
some convoluted expression in the SQL itself, so put the following function,
which takes a date/time value and an integer value representing the number of
seconds per group, into a standard module:

Public Function GroupTime(SampleTime As Date, Interval As Integer) As Date

Dim i As Integer

' get remainder when seconds value of
' sample time divided by interval
i = Second(SampleTime) Mod Interval

' if remainder = 0 then sample time is
' end of group so set return value
' to sample time, otherwise return
' end time for group in which sample
' time falls
If i = 0 Then
GroupTime = SampleTime
Else
GroupTime = DateAdd("s", Interval - i, SampleTime)
End If

End Function

It can then be called in a query like so:

PARAMETERS [Enter interval in seconds:] SHORT;
SELECT
GROUPTIME([SampleDate] + [SampleTime], [Enter interval in seconds:])
AS [Sample Date/Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
GROUP BY GROUPTIME([SampleDate] + [SampleTime], [Enter interval in seconds:]);

By entering 5 at the parameter prompt you should get a result set grouped by
each five second range. Entering 3 should give you it by three second
ranges. You can of course restrict the result to rows within a date range if
necessary by means of a WHERE clause as in my earlier examples.

Should the data types be text rather than date/time instead of [SampleDate]
+ [SampleTime] you'd use CDATE([SampleDate] & " " & Left([SampleTime],8)).

Ken Sheridan
Stafford, England

tobesus said:
Sorry Ken, let me try and do a better job of explaining what I want. Right
now I have one data point per second, and what I want to end up with is one
data point per 5 seconds - which would be the average (and stDev) of the last
5 seconds of data.

So if my table contained ten seconds of data like this:
SampleDate SampleTime XData
12/19/2006 21:36:01 0.028
12/19/2006 21:36:02 0.026
12/19/2006 21:36:03 0.026
12/19/2006 21:36:04 0.021
12/19/2006 21:36:05 0.026
12/19/2006 21:36:06 0.026
12/19/2006 21:36:07 0.026
12/19/2006 21:36:08 0.024
12/19/2006 21:36:09 0.024
12/19/2006 21:36:10 0.024

Then I would like my query to return two rows of data (one every 5 seconds)
like this:

SampleDate SampleTime Avg StDev
12/19/2006 21:36:05 0.0254 0.002608
12/19/2006 21:36:10 0.0248 0.001095

-where the average and standard deviation for the first five seconds are
represented by the first row of the query, and the average and standard
deviation for the next five seconds are represented by the second row, and so
on.

I would also like to be able to average over other intervals like 3s or 10s,
etc., which I'd imagine is just a matter of number subsitution in the SQL
once we get that worked out.

So if I have 100 seconds of table data, averaging at 5s intervals, I should
end up with 20 rows returned in my query. The queries I've tried so far have
all returned just one row of data per query.

Thanks again for your help and patience!


Ken Sheridan said:
What do you actually want? You can restrict the query over a date range,
which will return a single row:

PARAMETERS
[Enter start date:] DATETIME,
[Enter end date:] DATETIME;
SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([SampleTime]) MOD 5 = 0
AND [SampleDate] BETWEEN
[Enter start date:] AND [Enter end date:];

Or you can group it by a time period, e.g. month, which will return one row
per month:

PARAMETERS
[Enter year:] SHORT,
[Enter month:] SHORT;
SELECT
YEAR([SampleDate]) AS [Sample Year],
MONTH([SampleDate] AS [Sample Month],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([SampleTime]) MOD 5 = 0
AND YEAR([SampleDate]) = [Enter year:]
AND MONTH([SampleDate]) = [Enter month:]
GROUP BY YEAR([SampleDate]), MONTH([SampleDate]);

You could of course combine the two, restricting the query by a date range
and grouping within that range.

Both of the above queries use parameters which will prompt you to enter the
values when you run the queries. I've assumed the columns are of date/time
data type. If they are text data type then to return the year and month the
corresponding string expressions to YEAR([SampleDate]) and
MONTH([SampleDate]) would be LEFT([SampleDate],4] and MID(SampleDate],6,2)
and the parameters for these would declared as TEXT(4) and TEXT(2) rather
than as SHORT. In the first query the parameters would, if the column is
text, be declared as TEXT(10) and the values entered in exactly the same
format as the data in the table (be careful here that the actual values are
not in fact like 20061002 and what you are seeing is a formatted value).

Ken Sheridan
Stafford, England

tobesus said:
Thanks again Ken. Unfortunately with this new query I'm only getting back one
single row: the average and st dev of ALL the data apparently.

When I look at my table in design view, my SampleDate and SampleTime fields
have a data type listing of Date/Time, but both of the queries you suggested
produced the same result.

Any idea what the problem could be? Thanks again!

:

Mea culpa, I wasn't thinking straight. The query shouldn't return the
individual date/time values at all, just restrict the set over which the
average and standard deviation functions operate to those values, i.e.

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0;

Its not the format of the values which is important, but the data type of
the columns in the table. Are you sure they are of date/time data type. If
they are then the query would I think still be restricted to the correct rows
using the expression I gave you, but that would be fortuitous because of the
default format in which the value is shown (its actually a 64 bit floating
point number under the skin). If they really are of date/time data type a
better way would be to use the Second function:

SELECT
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE SECOND([Time]) MOD 5 = 0;

Ken Sheridan
Stafford, England

:

Thanks Ken. I changed my field names to SampleDate and SampleTime and ran the
query you recommended and it came very close to working.

The query parsed the data into five second intervals as intended, but didn't
actually average the data over five seconds. Instead it averaged the data
over one data point (every fifth second) and therefore produced no standard
deviation values because there was no deviation.

I noticed in my original table that my SampleDate and SampleTime fields were
in Date/Time format, not Text format as you were assuming, and I'm wondering
if that's the problem.

I tried changing them to Text format and tried to save; Access would work on
this for a few seconds and then report that "Access cannot change the data
type. There isn't enough disk space or memory."

I'm working with 1.5 million rows of data (x 3 fields columns) and am using
a 2-yr old Toshiba laptop with 1.23 GB of DDR SDRAM.

Any thoughts? Thanks again for your help!

:

I'm a assuming your Date and Time columns are of text data type rather than a
formatted date/time or number data type.

You can use the Mid function to extract the seconds value form your Time
column's values and then by means of the Mod operator restrict the query to
rows per n seconds, so to return rows at 5 second intervals a query would
look like this,

SELECT [Date], [Time],
AVG(Xdata) AS Average,
STDEV(Xdata) AS [Standard Deviation]
FROM YourTable
WHERE MID([Time], 7, 2) MOD 5 = 0
GROUP BY [Date], [Time];

I'd recommend that you avoid Date and Time as column names, however, as
these are the names of built in functions. Wrapping the name in square
brackets should avoid any confusion, but names such as SampleDate, SampleTime
would be better.

Incidentally you should be able to return a true single date/time data type
value from your two text expressions with:

CDate([Date]) + CDate(Left([Time],8))

This would enable you to do date/time arithmetic on the returned values if
necessary, using the normal date/time functions available in Access.

Ken Sheridan
Stafford, England

:


Hi, I'm trying to analyze one simple column of data collected at 1-second
intervals over several weeks. Raw data I'm importing into Access look like
this:

DATE, TIME, XDATA
2006-10-02,11:15:45.000,1717
2006-10-02,11:15:46.000,1715
2006-10-02,11:15:47.000,1713
2006-10-02,11:15:48.000,1714
2006-10-02,11:15:49.000,1717

I'm trying to create a query that would give me the average and standard
deviation of the data (XDATA) over various time intervals. Ie. I may want to
average over 5-second intervals, or I may want to average over 10-intervals.

Ideally my query result would have two data colums (AVERAGE and STDEV), and
would also have two additional columns for the date and time that the average
refers to.

Seems like this should be fairly easy to do but I'm still learning. Any help
would be greatly appreciated!
 
K

Ken Sheridan

It occurred to me that the function would require the interval to be a factor
of 60 as it works on the basis of the seconds value only. This is fine with
ranges up to 6 seconds, and with 10, 12, 15, 20 and 30 second ranges, but
with other intervals, e.g. 7 seconds, would break down. The following
amended function should overcome this, by using the full date/time value from
01/01/2000 (to avoid an overflow resulting from starting from Access's day
zero on 30 December 1899), i.e. there is one origin for the ranges starting
at one minute after midnight on 01/01/2000 rather than having new origins at
the start of each minute throughout the day. This means it will only work
with values from 01/01/2000 00:00:01 onwards of course:

Public Function GroupTime(SampleTime As Date, Interval As Long) As Date

Const STARTOF2000 = 36526
Const SECONDSINDAY = 86400
Dim l As Long

' get remainder when value of sample date/time
' from 01/01/2000 in seconds divided by interval
l = ((SampleTime - STARTOF2000) * SECONDSINDAY) Mod Interval

' if remainder = 0 then sample time is
' end of group so set return value
' to sample time, otherwise return
' end time for group in which sample
' time falls
If l = 0 Then
GroupTime = SampleTime
Else
GroupTime = DateAdd("s", Interval - l, SampleTime)
End If

End Function

Ken Sheridan
Stafford, England
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top