Appearance
Collecting data through pull ingress
Pull ingress lets NOOSAdmin retrieve information from external sources. It is useful when a source makes data available for collection, such as an HTTP response or a periodically published file.
Retrieval can be initiated by a recurring rule or a system-wide trigger. Pull ingress also supports scripted fetching, authenticated calls, configurable retries and throttling, allowing the collection process to account for the source's access and timing requirements.
Sources and formats
The documented retrieval options include HTTP call results and data from CSV or ZIP files obtained through FTP, HTTP or blob storage.
| Source or delivery method | Example use |
|---|---|
| HTTP call results | Retrieve information returned by an external service. |
| Files obtained through HTTP | Download a published CSV export or ZIP file. |
| Files obtained through FTP | Collect a file made available on an FTP server. |
| Files obtained through blob storage | Retrieve a file held in an external storage location. |
CSV represents tabular information as text. ZIP is an archive format used to package files. The contents and structure expected by a particular retrieval process must be agreed with the source.
These options describe how information can be obtained. The integration also needs to establish how that information corresponds to the application's destination data model.
Scripted retrieval
Some sources require more than a single request to obtain the desired data. NOOSAdmin supports scripts that automate the retrieval sequence.
For example, a source might publish a different export URL each day. A script could first make an HTTP call to discover the latest URL and then use that URL to fetch the data. This avoids relying on an unchanging file address when the source uses a discovery step.
The script implements the retrieval logic required by that source. This guide describes its purpose without prescribing a scripting interface or a particular external service contract.
Authenticated requests
Pull ingress can make authenticated calls. This supports sources that require credentials or another authentication arrangement before allowing access to their information.
The relevant method and access details depend on the source and integration. Planning retrieval therefore includes identifying how NOOSAdmin will be authorised to obtain the data, alongside the location and format of that data.
Schedules and triggers
Pulling can start automatically through recurring rules, such as an interval expressed in minutes or hours, or after a system-wide trigger fires.
A schedule is useful for checking a source at regular intervals. A trigger is useful when collection should be initiated in response to an event. The choice depends on when the source makes information available and what should cause the application to retrieve it.
Retries and throttling
Retries and throttling address distinct collection concerns.
Retries allow fetching to be attempted again if it fails, up to a configurable number of retries. This can help a collection process tolerate a temporary retrieval problem. A retry limit bounds the repeated attempts; it does not guarantee that data will ultimately be obtained.
Throttling enforces a minimum elapsed time between successive calls associated with the same trigger. This can limit how frequently a particular collection operation calls its source.
A schedule determines when recurring retrieval should start. Throttling constrains call frequency, while retries address failed fetching attempts. Their detailed interaction, including retry timing and how closely spaced invocations are handled, depends on the implementation and configuration.
These are documented pull-ingress capabilities. They should not be assumed to apply automatically to processors, runners or a complete multi-operation workflow.
Example: collecting a periodic measurement export
In an illustrative equipment application, a source publishes a CSV export and provides its current location through an HTTP response. NOOSAdmin is configured to retrieve the export periodically.
| Part of the collection | Illustrative arrangement |
|---|---|
| Initiation | A recurring rule starts retrieval. |
| Discovery | A script requests the current export URL. |
| Fetching | The script obtains the file from that URL. |
| Access | Calls use the source's required authentication arrangement. |
| Retrieval controls | Retry and throttling settings reflect the collection requirements. |
| Destination | Retrieved readings are mapped to the application's measurement data. |
Once the collected data is stored, users can explore it or use it as input to configured processing. The source's publication frequency should inform the collection interval: repeatedly retrieving an unchanged export may not provide new information. How repeated records are handled must be established for the integration.