-
Notifications
You must be signed in to change notification settings - Fork 23
Observer Manipulation
jojenki edited this page Jun 12, 2012
·
33 revisions
This feature is still experimental. Its definition may change at any time, without notice, and the data collected by it may be deleted on the development machines at any time.
An observer is a set of streams that define sets of data collected. This is analogous to our campaigns, where a campaign defines an overall set of surveys for a user.
A stream is a definition of a set of data to be collected. This is analogous to our surveys, where a survey defines a specific set of questions to ask a user.
Presently, all of our observers are defined in XML and the schema of each component is an Avro definition. The XML should have a root tag that is "observer" and is defined as follows:
| Key | Description | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | A unique ID for the observer. This should follow the Java naming conventions with periods as dividers between parts of the name, e.g. "org.ohmage.my.probe". It must begin with an alphanumeric character and may contain any combination of alphanumeric characters including periods to use as a separator; however, two periods may not appear together. The maximum length is 255 characters. | ||||||||||||||
| version | A number describing this observer's version. This value must be increased any time any part of the observer's definition is changed. | ||||||||||||||
| name | A user-friendly name for this probe. | ||||||||||||||
| description | A user-friendly description describing the overall purpose of this probe. Each stream will have its own description, which is the more applicable place to describe exactly what type of data is being collected. | ||||||||||||||
| versionString | A user-friendly string representing the version of the probe. | ||||||||||||||
| stream | Any number of these tags may exist as long as there is at least one. These are the individual streams of data being collected. They are defined as follows:
|
<?xml version="1.0" encoding="UTF-8"?>
<observer>
<id>edu.ucla.cens.Mobility</id>
<version>2012050700</version>
<name>Mobility</name>
<description>The Mobility probe collects the user's current movement type (still, walking, running, etc.), which we call "mode", and may also collect the phone's movements.</description>
<versionString>3.0</versionString>
<stream>
<id>regular</id>
<version>2012050700</version>
<name>Mobility - Regular</name>
<description>This records only the user's mode.</description>
<metadata>
<timestamp />
<location />
</metadata>
<schema>
{
"type":"record",
"namespace":"edu.ucla.cens.MobilityRegular",
"name":"regular",
"doc":"Only contains the user's mode.",
"fields":[
{
"name":"mode",
"doc":"The user's mode.",
"type":"string"
}
]
}
</schema>
</stream>
<stream>
<id>extended</id>
<version>2012050700</version>
<name>Mobility - Extended</name>
<description>This records the user's mode as well as accelerometer, WiFi, and GPS data.</description>
<metadata>
<timestamp />
<location />
</metadata>
<schema>
{
"type":"record",
"namespace":"edu.ucla.cens.MobilityExtended",
"name":"extended",
"doc":"Only contains the user's mode.",
"fields":[
{
"name":"mode",
"doc":"The user's mode.",
"type":"string"
},
{
"name":"speed",
"doc":"The user's speed in the last minute.",
"type":"double"
},
{
"name":"accel_data",
"doc":"An array of the accelerometer readings over the last minute.",
"type":{
"type":"array",
"items":{
"type":"record",
"name":"accel_point",
"doc":"A single accelerometer reading.",
"fields":[
{
"name":"x",
"doc":"The x-component of the accelerometer reading.",
"type":"double"
},
{
"name":"y",
"doc":"The y-component of the accelerometer reading.",
"type":"double"
},
{
"name":"z",
"doc":"The z-component of the accelerometer reading.",
"type":"double"
}
]
}
}
},
{
"name":"wifi_data",
"doc":"A WiFi reading from the last minute.",
"type":{
"type":"record",
"name":"wifi_point",
"doc":"A WiFi reading from the last minute.",
"fields":[
{
"name":"time",
"doc":"The time this data was recorded.",
"type":"long"
},
{
"name":"timezone",
"doc":"The time zone of the device when this data was recorded.",
"type":"string"
},
{
"name":"scan",
"doc":"The scan of WiFi information.",
"type":{
"type":"array",
"items":{
"type":"record",
"name":"access_point",
"doc":"A single access point's information.",
"fields":[
{
"name":"ssid",
"doc":"The access point's SSID.",
"type":"string"
},
{
"name":"strength",
"doc":"The strength of the signal from the access point.",
"type":"double"
}
]
}
}
}
]
}
}
]
}
</schema>
</stream>
</observer>
Allows a user to create a new observer.
observer/create
Anyone is allowed to create observers.
- (r) user = The user's username.
- (r) password = The user's password.
OR
- (r) auth_token = The user's authentication token.
- (r) client = A description of the client making the call.
- (r) observer_definition = The observer's definition as defined above.
POST /app/observer/create HTTP/1.1
Host: dev.ohmage.org
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7
Content-Length: byte-length-of-content
Content-Type: application/x-www-form-urlencoded
auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e
&observer_definition=<XML Content>
curl -v -F "auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e" -F "observer_definition=@/myObserver.xml;type=text/xml" http://localhost:8080/app/observer/create
{
"result" : "success"
}
See the error page for a description of error codes and their associated descriptions.
Uploads a set of data points for a user. All of the points must belong to the same observer but may belong to any of the streams and any of those streams' versions associated with that observer. The idea is that a data point is first associated with an observer, to which all of the data points in a single upload must belong. Then, it is associated with a stream for any version of that observer. Then, it is associated with a specific version of that stream, which defines a specific schema.
stream/upload
Anyone is allowed to upload their own data to an existing observer.
- (r) user = The user's username.
- (r) password = The user's password.
OR
- (r) auth_token = The user's authentication token.
- (r) client = A description of the client making the call.
- (r) observer_id = The unique identifier for this observer.
- (r) observer_version = The current version of this observer that is being used to generate this data. This is required as it is needed for the stream/read API. If a maintainer were to update an observer but not modify one of the existing streams, then, without this parameter, it would be impossible to facilitate stream/read's "observer_version" parameter which allows a caller to limit the data to only those points after the change.
- (r) data = The data to be uploaded. This have the format described below.
The data must be a JSON array of JSON objects. Each object must follow this definition:
| Key | Description | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| stream_id | The unique identifier for the stream to which this data applies. | ||||||||||||||||||||||||
| stream_version | The version of this stream to which this data applies. | ||||||||||||||||||||||||
| metadata | This should be a JSON object containing the metadata for this point. This field is optional, even if the definition states that records may contain metadata. The format of this object is:
|
||||||||||||||||||||||||
| data | The data according which must match this stream's definition. Additional fields may be provided, but the defined fields must be present. The additional fields will be saved. |
POST /app/stream/upload HTTP/1.1
Host: dev.ohmage.org
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7
Content-Length: byte-length-of-content
Content-Type: application/x-www-form-urlencoded
auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e
&observer_id=org.ohmage.myObserver
&observer_version=123456789
&data=<The data>
curl -v -F "auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e" -F "observer_id=org.ohmage.myObserver" -F "observer_version=123456789" -F "data=@/my.data;type=text/xml" http://localhost:8080/app/stream/upload
{
"result" : "success"
}
See the error page for a description of error codes and their associated descriptions.
Reads the contents of a stream. For now, we only allow a single stream's data to be read at a time. The reason for this is that all of the parameters are specific to each stream read, so allowing multiple streams to be read simultaneously would require that the stream ID, stream version, and column list would need to be duplicated for each stream.
stream/read
Anyone is allowed to read their own data. For now, we don't have a way to allow users to read data about other users.
- (r) user = The user's username.
- (r) password = The user's password.
OR
- (r) auth_token = The user's authentication token.
- (r) client = A description of the client making the call.
- (r) observer_id = The unique identifier for this observer.
- (o) observer_version = The version of the observer. Because a stream may not change between observer version changes, then querying a stream with a specific version and not providing this parameter would allow you to read across observer versions where the stream version didn't change.
- (r) stream_id = The unique identifier for the steam.
- (r) stream_version = The version of the stream.
- (o) start_date = Limits the results to only those after or on this date. This must be an ISO8601 date-time of the form "yyyy-MM-dd'T'HH:mm:ss.SSSZZ".
- (o) end_date = Limits the results to only those before or on this date. This must be an ISO8601 date-time of the form "yyyy-MM-dd'T'HH:mm:ss.SSSZZ".
- (o) column_list = Limits the results to only those whose columns are in this list. The format of the list is a comma-separated set of strings. Each string denotes the column to return based on the following schema, <column>:<sub-column>:<sub-sub-column>:.... For a record of the form {"a":"b","one":{"sub1":"first", "sub2":"second", "sub3":[{"sub-sub1":1, "sub-sub2":2}, {"sub-sub1":10, "sub-sub3":30}]}}, the column_list a, one:sub1, one:sub3:sub-sub2 would return {"a":"b", "one":{"sub1":"first", "sub3":[{"sub-sub2":2}, {}]}}. The columns do not need to be defined in the schema, so additional data that was added may be queried, but records without that data will be empty but present, i.e. {}.
- (o) num_to_skip = The number of records to skip. If this is negative, it will be set to 0. It is valid to skip more records than exist, but the number of records returned will be 0.
- (o) num_to_return = The number of records to return. If this is negative or greater than the maximum, it will be reset to the maximum. The current maximum is 2000.
POST /app/stream/read HTTP/1.1
Host: dev.ohmage.org
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7
Content-Length: byte-length-of-content
Content-Type: application/x-www-form-urlencoded
auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e
&observer_id=org.ohmage.myObserver
&stream_id=myStream
&stream_version=123456789
&column_list=some_int,some_record:sub_item,some_array:sub_item:sub_sub_item
curl -v -F "auth_token=abcfcd36-ab25-4494-8434-7798cb1d718e" -F "observer_id=org.ohmage.myObserver" -F "stream_id=myStream" -F "stream_version=123456789" -F "column_list=some_int,some_record:sub_item,some_array:sub_item:sub_sub_item" http://localhost:8080/app/stream/read
{
"result" : "success"
"metadata":{
"count":<A number representing the number of results.>,
"prev":"<The URL for the previous set of results.>",
"next":"<The URL for the next set of results.>"
},
"data":[
{
"metadata":{ // If no metadata exists, this key will not exist.
"timestamp":"<The ISO8601 timestamp.>" // If no timestamp exists, this key will not exist.
"location":{ // If no location exists, this key will not exist.
"latitude":12.345
"longitude":67.890
"accuracy":15
"provider":"Magic"
}
},
"data":{} // The data based on the given columns and the data collected for this point.
},
...
]
}
See the error page for a description of error codes and their associated descriptions.