Skip to main content

Scanners

info

Scanners allow you to scan and monitor your data sources for any changes or anomalies.

Scanners can be found in the Observability section of the menu bar.

The Scanners section in MINEO displays a table with information about all scanners created in the system. The table contains the following columns

  • Name: The name of the scanner.
  • Table: The name of the table the scanner is watching.
  • Description: A short description of the scanner and what it does.
  • Creator: The user that created the scanner.
  • Crontab expression: The crontab expression that defines the schedule for the scanner to run.
  • Crontab is active: A flag indicating whether the crontab expression is currently active or not.
  • Last execution status: The status of the last execution of the scanner, which can be Success, Failure or In Progress.
  • Snapshots: A count of the snapshots taken by the scanner. A snapshot is specific "anonymous picture" of your data at certain time.
  • Actions: The actions that can be performed on the scanner, including Edit, Run, and Delete.

The Scanners section provides a high-level view of the scanners in your system and the data they are monitoring. With the ability to view name, table, description, creator, crontab expression, and last run status, you can quickly see which scanners are active, when they were last run, and how they are performing.

How to create a scanner​

To create a new scanner press the plus button ➕ in the scanners table.

General configuration​

The available fields are:

  • Name: Give your scanner a descriptive name to help you identify it later.
  • Description: Enter a brief description of what the scanner is for, including any relevant information about the data source being monitored.
  • Data Source: Select the data source you want to monitor from the list of data sources available in MINEO.
  • Table: Specify the table within the data source you want to monitor.
  • Crontab expression: Use the crontab expression to specify how often you want the scanner to run. For example, if you want the scanner to run every day at 10:00 am, you would use the expression "0 10 * * *".
  • Excluded columns: Specify any columns within the table to be excluded from the scan. By default, all the columns are included.
  • Execution parameters: The execution parameters provides additional configuration options, like the maximum number of rows to process and the maximum time in seconds to process a table.

Once you have completed the configuration, click the Save button to save your changes. The scanner will then be added to the list of available scanners in MINEO and will run according to the schedule specified in the Crontab expression or manually choosing the scanner in the scanner's table and pressing the action run.

Checks​

This tab allows you to configure which checks are performed each time the scanner runs.

The available checks are

  • No New Data : Monitors for new data added to the table since the last scan.
  • New Data Amount Out of Forecast: Monitors for changes in the amount of new data added to the table compared to previous scans. You can adjust in the check a sensitivity threshold that defines how sensitive is the process.
  • Schema changed: Monitors for changes in the schema of the table, including changes in column names, data types, and other structural elements.
  • Amount of Primary Changes Out of Forecast: Monitors for changes in the number of primary key changes within the table compared to previous scans.
  • Column distribution changes: Monitors for changes in the distribution of values within a specific column, including changes in the number of unique values and the distribution of values. You can adjust in the check a sensitivity threshold that defines how sensitive is the process.

By using these options, you can gain a deeper understanding of your data sources and identify any changes or anomalies that may impact your analysis and insights.

SQL checks​

MINEO's SQL Checks feature allows you to monitor your data sources for specific conditions using custom SQL queries. Each SQL check consists of the following elements

  • Name: Give your SQL check a descriptive name to help you identify it later.
  • Description: Add a short description of what the SQL check is for, including any relevant information about the data source being monitored.
  • SQL Query: The SQL query that will be used to perform the check. The query should be written using standard SQL syntax.
  • Acceptance Condition: Specify the acceptance condition that determines whether the SQL check is considered passed or failed. The available options are
    • True if all results are 1 or True.
    • True if no rows are returned.
    • True if the results have at least one row.
    • True if the results have exactly one row.
    • True if all results are 0 or False.

Using the SQL Checks feature in MINEO, you can monitor your data sources for specific conditions such as data integrity, data consistency, or data completeness.

SQL variables​

SQL variables in MINEO allow you to add additional data to your scans, which can be useful for monitoring and tracking changes in your data sources over time.

Each SQL variable consists of the following elements

  • Name: Give your SQL variable a descriptive name to help you identify it later.
  • Description: Add a brief description of what the SQL variable is for, including any relevant information about the data source you are monitoring.
  • Data Source (optional): Select the data source that will be used to retrieve the data for the SQL variable. If the variable is used by a scanner, the scanner's data source is used by default.
  • SQL Query: The SQL query that will be used to retrieve the data for the SQL variable. The query should be written using standard SQL syntax.
  • Checks (optional): Specify validations to monitor the data retrieved by the SQL variable. Available validations include:
    • Minimum value: Specify a minimum value for the SQL variable.
    • Max Value: Specify a maximum value for the SQL variable.
    • Max Increase: Specify the maximum increase allowed for the SQL variable.
    • Max Decrease: Specify the maximum decrease allowed for the SQL variable.
    • Automatic Prediction: Use automatic prediction to set a range for the SQL variable. The sensitivity of the range is configurable.

By using SQL Variables in MINEO, you can monitor and track changes in your data sources over time and gain a deeper understanding of your data.

Executions​

The Executions tab in MINEO provides a tabular view of all scans performed by your scanners. This tab allows you to monitor and manage the execution of your scanners in real time.

The table in the Executions tab contains the following information for each scan

  • Status: The current status of the scan, which can be "Running", "Finished" or "Cancelled".
  • Number of rows detected: The number of rows detected by the scan.
  • Number of rows processed: The number of rows processed by the scan.
  • Start Date: The date and time the scan started.
  • Finish Date: The date and time when the scan ended.
  • Elapsed Time: The elapsed time during the scan.
  • Logs: A link to the logs of each individual run, which provide detailed information about the scan and its results.

In addition to viewing this information, you can perform the following actions for each scan

  • Cancel: Cancel the scan if it is in progress.
  • Delete: Delete the scan from the list of executions.

The Executions tab provides real-time insight into the performance of your scanners, making it easy to monitor and manage your scans over time. With the ability to view logs, the number of lines detected and processed, and the execution time, you can gain a better understanding of your data and the performance of your scanners.