Online Handwritten Character Recognition Using Lipi Toolkit

Handwritten character recognition (HCR) has become a key technology in many modern applications such as digital notetaking, form processing, and assistive devices. While the problem has been tackled for decades, the rise of webbased platforms demands solutions that can operate entirely in a browser, without relying on serverside processing. The Lipi Toolkitan opensource JavaScript libraryprovides exactly that: a clientside engine capable of recognizing handwriting on HTML canvas elements in real time.

Why Choose an Online Solution?

Traditional HCR pipelines often require heavy preprocessing, feature extraction, and classification stages that are executed on powerful backend servers. This approach introduces latency, dataprivacy concerns, and dependence on network connectivity. An online, clientside implementation offers several advantages:

• Instant feedback: Recognition results appear instantly as the user writes.
• Privacypreserving: No image data leaves the users device.
• Scalability: No server load to manage; the same code works for millions of users.
• Crossplatform: Works on desktops, tablets, and smartphones through any modern browser.

What Is Lipi Toolkit?

Lipi Toolkit (formerly Lipi.js) is a lightweight JavaScript library that implements a trained neural network for recognizing a wide range of handwritten characters, including:

• English alphabets (uppercase & lowercase)
• Digits 09
• Common symbols (e.g., +, -, =, @)
• Selected Unicode scripts (Hindi, Tamil, etc.)

The library is built on top of TensorFlow.js, allowing it to run entirely in the browser using WebGL for accelerated computation. Lipi provides a simple API to initialize a canvas, capture strokes, and obtain the predicted character.

Core Components of an Online HCR Page

1. Canvas for drawing an HTML5 <canvas> element where users write characters.
2. Event handling JavaScript listeners for mouse/touch events that record stroke data.
3. Preprocessing Normalization of the captured image (centering, resizing, grayscale conversion).
4. Model inference Feeding the processed image into Lipis neural network and getting a probability distribution.
5. Result display Showing the toppredicted character and confidence score to the user.

StepbyStep Implementation

1. Include Lipi and TensorFlow.js

<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.9.0"></script><script src="https://cdn.jsdelivr.net/npm/lipi-toolkit@2.1.0/dist/lipi.min.js"></script>

Both scripts are served from a CDN, keeping the page size small.

2. Create the Drawing Canvas

<canvas id="drawArea" width="300" height="300"        style="border:1px solid #ccc; border-radius:4px;"></canvas>

3. Capture User Strokes

Using simple mouse and touch listeners we store points in an array. The snippet below works for both desktop and mobile devices.

const canvas = document.getElementById('drawArea');const ctx = canvas.getContext('2d');let drawing = false;let points = [];// Start drawingfunction start(event) {    drawing = true;    points = [];    const pos = getPos(event);    points.push(pos);    ctx.beginPath();    ctx.moveTo(pos.x, pos.y);}// Continue drawingfunction move(event) {    if (!drawing) return;    const pos = getPos(event);    points.push(pos);    ctx.lineTo(pos.x, pos.y);    ctx.strokeStyle = '#000';    ctx.lineWidth = 8;    ctx.lineCap = 'round';    ctx.lineJoin = 'round';    ctx.stroke();}// End drawingfunction end() {    drawing = false;    ctx.closePath();    recognize(); // Trigger recognition after each stroke}// Utility to get cursor/touch position relative to canvasfunction getPos(e) {    const rect = canvas.getBoundingClientRect();    const touch = e.touches ? e.touches[0] : e;    return {        x: touch.clientX - rect.left,        y: touch.clientY - rect.top    };}// Attach listenerscanvas.addEventListener('mousedown', start);canvas.addEventListener('mousemove', move);canvas.addEventListener('mouseup', end);canvas.addEventListener('mouseleave', end);canvas.addEventListener('touchstart', start);canvas.addEventListener('touchmove', move);canvas.addEventListener('touchend', end);

4. Preprocess the Canvas Image

Lipi expects a 2828 pixel grayscale image (the same size used for the MNIST dataset). The following function extracts the canvas content, resizes it, and normalizes pixel values.

function getImageData() {    // Create an offscreen canvas for resizing    const off = document.createElement('canvas');    off.width = off.height = 28;    const offCtx = off.getContext('2d');    // Fill background with white (important for contrast)    offCtx.fillStyle = '#fff';    offCtx.fillRect(0, 0, 28, 28);    // Draw the original canvas into the offscreen canvas, scaling down    offCtx.drawImage(canvas, 0, 0, 28, 28);    // Get pixel data and convert to a Float32Array normalized to [0,1]    const imgData = offCtx.getImageData(0, 0, 28, 28);    const data = imgData.data;    const gray = new Float32Array(28 * 28);    for (let i = 0, j = 0; i < data.length; i += 4, j++) {        // Convert RGB to luminosity; canvas is black on white, so invert        const lum = (0.299 * data[i] + 0.587 * data[i+1] + 0.114 * data[i+2]) / 255;        gray[j] = 1 - lum; // Invert: dark strokes become high values    }    return tf.tensor(gray, [1, 28, 28, 1]);}

5. Run Inference with Lipi

Lipi provides a predict method that returns an object containing the predicted character and confidence.

async function recognize() {    const inputTensor = getImageData();    // Lipi's model is loaded automatically on first call    const result = await Lipi.predict(inputTensor);    displayResult(result);    // Clean up tensors to avoid memory leaks    tf.dispose(inputTensor);}

6. Show the Result

function displayResult(res) {    const out = document.getElementById('output');    out.innerHTML = `Prediction: ${res.char}                       Confidence: ${(res.probability*100).toFixed(2)}%`;}

Complete HTML Example

The following snippet combines all the pieces into a single, functional page. Copy it into an .html file and open it with any modern browser.

<!DOCTYPE html><html lang="en"><head>    <meta charset="UTF-8">    <title>Handwritten Character Recognition with Lipi</title>    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.9.0"></script>    <script src="https://cdn.jsdelivr.net/npm/lipi-toolkit@2.1.0/dist/lipi.min.js"></script>    <style>        body{font-family:Arial,Helvetica,sans-serif;margin:20px;background:#f9f9f9;color:#333}        #drawArea{border:1px solid #ccc;border-radius:4px;cursor:crosshair}        #output{margin-top:10px;font-size:1.2em}    </style></head><body>    <h1>Online Handwritten Character Recognition</h1>    <canvas id="drawArea" width="300" height="300"></canvas>    <div id="output">Write a character above</div>    <script>        const canvas = document.getElementById('drawArea');        const ctx = canvas.getContext('2d');        let drawing = false;        let points = [];        function getPos(e){            const rect = canvas.getBoundingClientRect();            const touch = e.touches ? e.touches[0] : e;            return {x:touch.clientX-rect.left, y:touch.clientY-rect.top};        }        function start(e){ drawing=true; points=[]; const p=getPos(e); points.push(p); ctx.beginPath(); ctx.moveTo(p.x,p.y); }        function move(e){ if(!drawing) return; const p=getPos(e); points.push(p); ctx.lineTo(p.x,p.y); ctx.strokeStyle='#000'; ctx.lineWidth=8; ctx.lineCap='round'; ctx.lineJoin='round'; ctx.stroke(); }        function end(){ drawing=false; ctx.closePath(); recognize(); }        canvas.addEventListener('mousedown',start);        canvas.addEventListener('mousemove',move);        canvas.addEventListener('mouseup',end);        canvas.addEventListener('mouseleave',end);        canvas.addEventListener('touchstart',start);        canvas.addEventListener('touchmove',move);        canvas.addEventListener('touchend',end);        function getImageData(){            const off=document.createElement('canvas');            off.width=off.height=28;            const offCtx=off.getContext('2d');            offCtx.fillStyle='#fff';            offCtx.fillRect(0,0,28,28);            offCtx.drawImage(canvas,0,0,28,28);            const img=offCtx.getImageData(0,0,28,28);            const data=img.data;            const gray=new Float32Array(28*28);            for(let i=0,j=0;i${result.char}  Confidence: ${(result.probability*100).toFixed(2)}%`;            tf.dispose(tensor);        }    </script></body></html>

Extending the Basic Demo

The minimal example above can be enriched in many ways:

• Multiple character input: Detect word boundaries and run recognition on each segment.
• Language support: Load additional Lipi models for Devanagari, Tamil, or custom datasets.
• Feedback loop: Allow users to correct wrong predictions, then finetune the model locally with tfjs creating a personalized recognizer.
• Accessibility: Provide a button to clear the canvas and a voiceover suggestion of the recognized character.
• Performance monitoring: Display inference time to illustrate the efficiency of WebGL acceleration.

Best Practices & Considerations

1. Canvas size vs. model input Keep the drawing area reasonably large (e.g., 300px) so users can write comfortably, but always downscale to 28px before feeding the model.
2. Normalization Inverting colors (white background, black ink) aligns with the training data used by Lipi.
3. Touch handling Prevent scrolling while drawing on mobile by calling event.preventDefault() inside the touch listeners.
4. Memory management Dispose TensorFlow tensors after each prediction to avoid memory leaks, especially on long sessions.
5. Model loading Lipi loads its model lazily on the first call; show a spinner if you anticipate a noticeable delay on slower connections.

Conclusion

Lipi Toolkit makes it remarkably easy to embed handwritten character recognition directly into a web page. By leveraging TensorFlow.js, the heavy lifting of neuralnetwork inference happens on the client, giving users instant feedback while keeping their data private. The example provided demonstrates a clean, lightweight implementation that can serve as a foundation for more sophisticated applications such as notetaking apps, educational tools, or multilingual input methods.

With a few enhancementssupport for multiple scripts, realtime confidence visualizations, or userdriven model adaptationdevelopers can build robust, productionready HCR experiences using only standard web technologies.