Industrial Emergency Detection System

Solution

AI ​​application development was delegated to another team. Our responsibility was to develop a web application that could receive a video stream from surveillance cameras, redirect individual frames to the AI application (for further processing), and receive alarms and other data from it whenever incidents are detected. The system functions as follows:

The customer was already using a production management solution within the factory. It was a web application created in Jmix for overseeing, managing, and planning production operations. The customer did not want to create a standalone application for the emergency detection system, so we decided to integrate the solution under development into the existing Jmix application.

Development of an additional module for the Jmix web application

We developed an additional module for the customer's application in Java using the Jmix framework. The Jmix module allows users to configure the AI app. It also records and stores video files, sends incident notifications, and allows for viewing live video streams.

The AI and web applications are located on different servers. To configure the AI app, the user inputs its address and port in the web application interface. After availability verification, one can change the settings. For the initial setup, the web application generates a token and a link for the AI application to send incident data via HTTP POST requests.

After the initial setup, users can connect IP cameras by inputting a link to the RTSP stream, along with user-visible tags and descriptions. Browser notifications can be configured based on user roles. When receiving such a notification, the user can launch a new tab with the video stream from the corresponding camera. One can also connect SMS and email notifications for different incident types. 

The Jmix application captures the RTSP stream from IP cameras and records it as video files with a maximum duration of 1 hour. The information about the camera (tags and description) and the video file (duration, resolution, and other technical details) is saved in the database, while the video itself is saved as an mp4 file in the local file storage of the application. Video files are preserved for 1 week unless linked to an incident, in which case deletion requires specific user permissions.

The Jmix application sends individual frames of the video stream to the AI ​​app. When an emergency is detected, the latter sends an alarm signal and accompanying incident details to the Jmix application:

• Camera ID
• Event timestamp
• Event type 
• Frame with a bounding box encapsulating the detected incident area

When an alarm is received, the Jmix application displays a notification in the browser for designated user groups and dispatches SMS and email notifications based on configured settings. Clicking the “See details” button launches a new tab with a live video stream from the camera that detected the incident and a static frame highlighting the event. This way, users can quickly inspect the scene and ensure it isn’t a false alarm.

Video streams are accessible not only through notifications but also by navigating to the IP camera page for real-time viewing with a slight delay. To transmit video from cameras, the application segments the continuous video stream into smaller files that are delivered in response to browser requests. The segmentation process is initiated when at least one user is actively watching the camera's feed.

Major Issues Resolved

Embedding a video stream in the Jmix application

Video cannot be processed with the standard Jmix tools. Our only options were either to develop custom components or to use external ones. After research, the team decided to work with the VideoJS Player addon.

In this project, processing video streams follows a standardized algorithm:

1. The RTSP stream is captured.

2. A temporary file for the video is created.

3. The stream is saved to a temporary file.

4. The browser player loads the file.

Receiving video from the RTSP streams for the server

Neither VideoJS Player nor standard Jmix and Java tools offer direct support for processing RTSP streams from IP cameras. 

In this case, the solution can be found in the JavaCV framework that was used to implement the AI ​​application. It contains the FFMpeg library. Its tools (FrameGrabber and FrameRecorder) are used to capture and process individual frames, as well as record video.

To connect a new camera in the Jmix application, users specify the address of the RTSP stream, which is then used by FrameGrabber to capture the video stream on the server side.

The Scope of Work

• Jmix application module development. The module sends HTTP requests to the AI ​​application and provides an HTTP API, allowing users to view video from cameras online. It also sends browser, SMS, or email notifications to users and saves video files that can be downloaded for playback later.

Technologies Used

• The web application was developed in Java using the Jmix framework.
• Video transmission from RTSP IP cameras was implemented in the H.264 format.
• The video stream is captured using the JavaCV framework by Bytedeco.

Result

Integra Sources developed a custom module to enhance the customer's web application. The module receives video feeds from IP cameras located in workshop areas, enabling real-time viewing and server-side video storage. Individual frames are sent to the AI application that detects emergency incidents.

When an emergency is detected, the web application promptly dispatches browser, SMS, and email notifications to designated personnel. Operators can swiftly access live video feeds from the relevant IP camera to assess the situation and verify the alarm's validity.

The solution allowed the customer to:

• Boost occupational safety at the enterprise.
• Implement timely countermeasures or initiate evacuations in response to emergencies or accidents.
• Reduce potential property damage by quickly eliminating accidents.
• Minimize the risk of personal injury and subsequent financial liabilities.