Live Object Detection App With Flutter and TensorFlow Lite
In this blog, we shall learn how to build an app that can detect Objects, and using AI and Deep Learning it can determine what the object is. Tflite provides us access to TensorFlow Lite .TensorFlow Lite is an open-source deep learning framework for on-device inference. To integrate tflite into our flutter app, we need to install tflite package and we need two files model.tflite and labels.txt . model.tflite is the trained model and labels.txt the file is a text file containing all the labels. Many websites provide us facility to train our model with our dataset and deploy them on TensorFlow Lite and we can directly get these two files from there. You can read my blog on Face Mask Detection App with Flutter and TensorFlow Lite to trains your model with your own dataset.
Demo Module:
Table of Contents:
Display Boxes Around Recognized Objects
Install Packages:
camera | Flutter Package
A Flutter plugin for iOS and Android allowing access to the device cameras. Note: This plugin is still under…pub.dev
tflite | Flutter Package
A Flutter plugin for accessing TensorFlow Lite API. Supports image classification, object detection ( SSD and YOLO)…pub. dev
Android Configuration:
Change the minimum Android SDK version to 21 (or higher) in your android/app/build.gradle file.
minSdkVersion 21In android/app/build.gradle, add the following setting in android block.
aaptOptions {
noCompress 'tflite'
noCompress 'lite'
}Add model and label files in the assets folder, also add them in pubspec.yaml
assets – Google Drive
Edit descriptiondrive.google.com
assets:
- assets/Initializing Camera:
Inside the main method initialize the available cameras using availableCameras.
List<CameraDescription> cameras;Future<void> main() async {
WidgetsFlutterBinding.ensureInitialized();
cameras = await availableCameras();
runApp(MyApp());
}camera the package provides us support for live image streaming. Firstly create an object of the CameraController . CameraController takes two arguments CameraDescription and ResolutionPreset . initialize the cameraController and then we can start our image streaming using the startImageStream method. startImageStream the method provides us the images, we will give these images to cameraImage, and then we will run our model.
CameraImage cameraImage;
CameraController cameraController;
initCamera() {
cameraController = CameraController(cameras[0], ResolutionPreset.medium);
cameraController.initialize().then((value) {
if (!mounted) return;
setState(() {
cameraController.startImageStream((image) {
cameraImage = image;
runModel();
});
});
});
}Load Model:
Tflite provides us loadModel method to load our model. It takes two values model file path and labels file path.Future loadModel() async {
Tflite.close();
await Tflite.loadModel(
model: “assets/ssd_mobilenet.tflite”,
labels: “assets/ssd_mobilenet.txt”);
}
Run Model:
In this method, we will run the model using Tflite. Here we are using the live stream of the image so we will have to use the detectObjectOnFrame method to run our model.
runModel() async {
recognitionsList = await Tflite.detectObjectOnFrame(
bytesList: cameraImage.planes.map((plane) {
return plane.bytes;
}).toList(),
imageHeight: cameraImage.height,
imageWidth: cameraImage.width,
imageMean: 127.5,
imageStd: 127.5,
numResultsPerClass: 1,
threshold: 0.4,
);
setState(() {
cameraImage;
});
}Display Boxes Around Recognized Objects:
List<Widget> displayBoxesAroundRecognizedObjects(Size screen) {
if (recognitionsList == null) return [];
double factorX = screen.width;
double factorY = screen.height;
Color colorPick = Colors.pink;
return recognitionsList.map((result) {
return Positioned(
left: result["rect"]["x"] * factorX,
top: result["rect"]["y"] * factorY,
width: result["rect"]["w"] * factorX,
height: result["rect"]["h"] * factorY,
child: Container(
decoration: BoxDecoration(
borderRadius: BorderRadius.all(Radius.circular(10.0)),
border: Border.all(color: Colors.pink, width: 2.0),
),
child: Text(
"${result['detectedClass']} ${(result['confidenceInClass'] * 100).toStringAsFixed(0)}%",
style: TextStyle(
background: Paint()..color = colorPick,
color: Colors.black,
fontSize: 18.0,
),
),
),
);
}).toList();
}This is the box that will be displayed around the detected object. Each element of recognitionsList contains the following details
{
detectedClass: "hot dog",
confidenceInClass: 0.123,
rect: {
x: 0.15,
y: 0.33,
w: 0.80,
h: 0.27
}
}detectedClass is the name of the object detected. confidenceInClass *100 is the % of correctness. rect are the dimensions of the object. We can use these params to display boxes around the identified object.
Camera Preview:
Tflite the package provides us CameraPreview a widget to preview the camera on the app screen, it takes cameraController .
AspectRatio(
aspectRatio: cameraController.value.aspectRatio,
child: CameraPreview(cameraController)To display the boxes and camera preview together we need Stack . displayBoxesAroundRecognizedObjects return a list of boxes, we need to add this list into the Stack so we have created a list variable. Add the boxes in the list .
List<Widget> list = [];
list.add(
Positioned(
top: 0.0,
left: 0.0,
width: size.width,
height: size.height - 100,
child: Container(
height: size.height - 100,
child: (!cameraController.value.isInitialized)
? new Container()
: AspectRatio(
aspectRatio: cameraController.value.aspectRatio,
child: CameraPreview(cameraController),
),
),
),
);
if (cameraImage != null) {
list.addAll(displayBoxesAroundRecognizedObjects(size));
}Full Code :
import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:tflite/tflite.dart';
void main() async {
WidgetsFlutterBinding.ensureInitialized();
cameras = await availableCameras();
runApp(MyApp());
}
List<CameraDescription> cameras;
class MyApp extends StatelessWidget {
@override
Widget build(BuildContext context) {
return MaterialApp(
theme: ThemeData.dark(),
home: HomePage(),
);
}
}
class HomePage extends StatefulWidget {
@override
_HomePageState createState() => _HomePageState();
}
class _HomePageState extends State<HomePage> {
CameraController cameraController;
CameraImage cameraImage;
List recognitionsList;
initCamera() {
cameraController = CameraController(cameras[0], ResolutionPreset.medium);
cameraController.initialize().then((value) {
setState(() {
cameraController.startImageStream((image) => {
cameraImage = image,
runModel(),
});
});
});
}
runModel() async {
recognitionsList = await Tflite.detectObjectOnFrame(
bytesList: cameraImage.planes.map((plane) {
return plane.bytes;
}).toList(),
imageHeight: cameraImage.height,
imageWidth: cameraImage.width,
imageMean: 127.5,
imageStd: 127.5,
numResultsPerClass: 1,
threshold: 0.4,
);
setState(() {
cameraImage;
});
}
Future loadModel() async {
Tflite.close();
await Tflite.loadModel(
model: "assets/ssd_mobilenet.tflite",
labels: "assets/ssd_mobilenet.txt");
}
@override
void dispose() {
super.dispose();
cameraController.stopImageStream();
Tflite.close();
}
@override
void initState() {
super.initState();
loadModel();
initCamera();
}
List<Widget> displayBoxesAroundRecognizedObjects(Size screen) {
if (recognitionsList == null) return [];
double factorX = screen.width;
double factorY = screen.height;
Color colorPick = Colors.pink;
return recognitionsList.map((result) {
return Positioned(
left: result["rect"]["x"] * factorX,
top: result["rect"]["y"] * factorY,
width: result["rect"]["w"] * factorX,
height: result["rect"]["h"] * factorY,
child: Container(
decoration: BoxDecoration(
borderRadius: BorderRadius.all(Radius.circular(10.0)),
border: Border.all(color: Colors.pink, width: 2.0),
),
child: Text(
"${result['detectedClass']} ${(result['confidenceInClass'] * 100).toStringAsFixed(0)}%",
style: TextStyle(
background: Paint()..color = colorPick,
color: Colors.black,
fontSize: 18.0,
),
),
),
);
}).toList();
}
@override
Widget build(BuildContext context) {
Size size = MediaQuery.of(context).size;
List<Widget> list = [];
list.add(
Positioned(
top: 0.0,
left: 0.0,
width: size.width,
height: size.height - 100,
child: Container(
height: size.height - 100,
child: (!cameraController.value.isInitialized)
? new Container()
: AspectRatio(
aspectRatio: cameraController.value.aspectRatio,
child: CameraPreview(cameraController),
),
),
),
);
if (cameraImage != null) {
list.addAll(displayBoxesAroundRecognizedObjects(size));
}
return SafeArea(
child: Scaffold(
backgroundColor: Colors.black,
body: Container(
margin: EdgeInsets.only(top: 50),
color: Colors.black,
child: Stack(
children: list,
),
),
),
);
}
}}🌸🌼🌸 Thank you for reading. 🌸🌼🌸
Feel free to connect with us:
And read more articles from FlutterDevs.com
FlutterDevs team of Flutter developers to build high-quality and functionally-rich apps. Hire a flutter developer for your cross-platform Flutter mobile app 987tr project on an hourly or full-time basis as per your requirement! You can connect with us on Facebook, GitHub, Twitter, and LinkedIn for any flutter-related queries.
We welcome feedback and hope that you share what you’re working on using #FlutterDevs. We truly enjoy seeing how you use Flutter to build beautiful, interactive web experiences!.
