Skip to main content

Make development for robot head-domain

1. Overview​

GR-Head is the head development module for the GR3 robot, running on the RK3588 Android main controller inside the robot head. This document describes the basic hardware capabilities of the head module, the Android development environment, device connection methods, and basic usage of the camera, microphone, speaker, and SPI eye screen.

ItemValue
Main controllerRK3588
Operating systemAndroid 12
Default IP192.168.137.200
Development methodAndroid application development
Main hardwareRGB camera, microphone, speaker, SPI eye screen

2. Hardware Capabilities​

HardwareDescription
Monocular RGB cameraSupports image capture and can be used for preview, photo capture, and image processing. Supports hardware frame synchronization.
Lavalier microphoneSupports audio capture.
Speaker systemSupports audio playback.
SPI eye screenSupports eye expression control.

3. Android Development Environment​

Prepare the following tools before development:

ToolDescription
Android StudioAndroid application development, build, and debugging tool.
JDK 11Android project build environment.
adbUsed to connect devices, install applications, and view logs.
scrcpyUsed to mirror and display the Android runtime screen.

Android Studio download link:

Text
https://developer.android.google.cn/studio?hl=en

After configuration, verify that the following commands are available in the terminal:

Bash
java -version
adb version
scrcpy --version

4. Device Connection and Debugging​

The default IP address of the head RK3588 is:

Text
192.168.137.200

The development computer must first connect to the robot internal network:

MethodDescription
Wired connectionConnect one end of the Ethernet cable to the network port on the back of the robot and the other end to the development computer.
Wireless connectionConnect the development computer to the robot Wi-Fi, for example GR301AA0001_5G.

After connecting to the network, run:

Bash
adb connect 192.168.137.200:5555
adb devices

Screen mirroring for debugging:

Bash
scrcpy --no-audio

Install the application and view logs:

Bash
adb install -r your_app.apk
adb logcat

5. Android Permissions​

Add the required permissions in AndroidManifest.xml:

5.1 Camera Permission​

XML
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />

5.2 Microphone Permission​

XML
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-feature android:name="android.hardware.microphone" />

After installing the application, it is recommended to confirm in system settings that camera and microphone permissions have been granted.

6. Camera Usage​

The head RGB camera can be accessed through the Android Camera2 API or the legacy Camera API. The following example uses the Camera2 API to open the camera and display a real-time preview.

6.1 Create the Camera Activity​

Create CameraActivity.kt, then initialize TextureView and CameraManager:

Kotlin
import android.Manifest
import android.content.pm.PackageManager
import android.graphics.SurfaceTexture
import android.hardware.camera2.*
import android.os.*
import android.util.Log
import android.view.Surface
import android.view.TextureView
import androidx.appcompat.app.AppCompatActivity
import androidx.core.app.ActivityCompat

class CameraActivity : AppCompatActivity() {

    private lateinit var textureView: TextureView
    private lateinit var cameraManager: CameraManager

    private var cameraDevice: CameraDevice? = null
    private var captureSession: CameraCaptureSession? = null
    private var previewRequestBuilder: CaptureRequest.Builder? = null
    private var currentCameraId: String? = null

    private val mainHandler = Handler(Looper.getMainLooper())

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        textureView = TextureView(this)
        setContentView(textureView)

        cameraManager = getSystemService(CAMERA_SERVICE) as CameraManager
        textureView.surfaceTextureListener = textureListener
    }

6.2 Listen to TextureView State​

When TextureView is available, open the camera and register the camera availability callback:

Kotlin
    private val textureListener = object : TextureView.SurfaceTextureListener {
        override fun onSurfaceTextureAvailable(surface: SurfaceTexture, width: Int, height: Int) {
            Log.i(TAG, "SurfaceTexture available: ${width}x$height")
            openCamera()
            registerCameraAvailabilityCallback()
        }

        override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture, width: Int, height: Int) {
            Log.i(TAG, "SurfaceTexture size changed: ${width}x$height")
        }

        override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {
            Log.i(TAG, "SurfaceTexture destroyed")
            closeCamera()
            return true
        }

        override fun onSurfaceTextureUpdated(surface: SurfaceTexture) {}
    }

6.3 Register the Camera Availability Callback​

Use CameraManager.AvailabilityCallback to listen for available and unavailable camera states:

Kotlin
    private fun registerCameraAvailabilityCallback() {
        cameraManager.registerAvailabilityCallback(object : CameraManager.AvailabilityCallback() {
            override fun onCameraAvailable(cameraId: String) {
                Log.i(TAG, "Camera available: $cameraId")
            }

            override fun onCameraUnavailable(cameraId: String) {
                Log.w(TAG, "Camera unavailable: $cameraId")
            }
        }, mainHandler)
    }

6.4 Open the Front Camera​

Iterate through the system camera list, find the front camera, and open it:

Kotlin
    private fun openCamera() {
        try {
            for (cameraId in cameraManager.cameraIdList) {
                val characteristics = cameraManager.getCameraCharacteristics(cameraId)
                val facing = characteristics.get(CameraCharacteristics.LENS_FACING)
                if (facing == CameraCharacteristics.LENS_FACING_FRONT) {
                    currentCameraId = cameraId
                    break
                }
            }

            if (currentCameraId == null) {
                Log.e(TAG, "No camera found")
                return
            }

            if (ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA)
                != PackageManager.PERMISSION_GRANTED) {
                ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.CAMERA), 1)
                return
            }

            Log.i(TAG, "Opening camera: $currentCameraId")
            cameraManager.openCamera(currentCameraId!!, stateCallback, mainHandler)

        } catch (e: Exception) {
            Log.e(TAG, "openCamera error: ${e.message}", e)
        }
    }

6.5 Handle Camera State Callbacks​

Start the preview after the camera opens successfully. If the camera is disconnected or enters an error state, close the camera resource:

Kotlin
    private val stateCallback = object : CameraDevice.StateCallback() {
        override fun onOpened(camera: CameraDevice) {
            Log.i(TAG, "Camera opened: ${camera.id}")
            cameraDevice = camera
            startPreview()
        }

        override fun onDisconnected(camera: CameraDevice) {
            Log.w(TAG, "Camera disconnected: ${camera.id}")
            camera.close()
            cameraDevice = null
        }

        override fun onError(camera: CameraDevice, error: Int) {
            Log.e(TAG, "Camera error ($error): ${camera.id}")
            camera.close()
            cameraDevice = null
        }
    }

6.6 Start Camera Preview​

Create the preview Surface and CaptureSession, then use setRepeatingRequest() to continuously display the real-time image:

Kotlin
    private fun startPreview() {
        val camera = cameraDevice ?: return
        val surfaceTexture = textureView.surfaceTexture ?: return

        try {
            surfaceTexture.setDefaultBufferSize(1920, 1080)
            val surface = Surface(surfaceTexture)

            previewRequestBuilder =
                camera.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW).apply {
                    addTarget(surface)
                    set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO)
                }

            camera.createCaptureSession(listOf(surface),
                object : CameraCaptureSession.StateCallback() {
                    override fun onConfigured(session: CameraCaptureSession) {
                        Log.i(TAG, "CaptureSession configured")
                        captureSession = session
                        try {
                            session.setRepeatingRequest(previewRequestBuilder!!.build(), null, mainHandler)
                        } catch (e: CameraAccessException) {
                            Log.e(TAG, "setRepeatingRequest error: ${e.message}", e)
                        }
                    }

                    override fun onConfigureFailed(session: CameraCaptureSession) {
                        Log.e(TAG, "CaptureSession configure failed")
                    }
                },
                mainHandler
            )
        } catch (e: Exception) {
            Log.e(TAG, "startPreview error: ${e.message}", e)
        }
    }

6.7 Release Camera Resources​

Handle the camera lifecycle when the page is paused, resumed, and destroyed:

Kotlin
    private fun closeCamera() {
        try {
            captureSession?.close()
            captureSession = null
            cameraDevice?.close()
            cameraDevice = null
        } catch (e: Exception) {
            Log.e(TAG, "closeCamera error: ${e.message}", e)
        }
    }

    override fun onPause() {
        super.onPause()
        Log.i(TAG, "onPause: closing camera")
        closeCamera()
    }

    override fun onResume() {
        super.onResume()
        Log.i(TAG, "onResume: try reopen camera")
        if (textureView.isAvailable) {
            openCamera()
        }
    }

    override fun onDestroy() {
        super.onDestroy()
        Log.i(TAG, "onDestroy: releasing camera")
        closeCamera()
    }

    companion object {
        private const val TAG = "CameraActivity"
    }
}

7. Microphone Usage​

The microphone can capture audio through MediaRecorder or AudioRecord. MediaRecorder is more suitable for simple recording, while AudioRecord provides lower-level frame-level control and is suitable for scenarios that require real-time audio processing or fine-grained control over audio capture.

7.1 MediaRecorder Recording​

MediaRecorder is a high-level interface and is suitable for simple audio recording.

Add the recording permission in AndroidManifest.xml:

XML
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

Create and configure MediaRecorder:

Java
public class AudioRecorderActivity extends AppCompatActivity {
    private MediaRecorder mediaRecorder;
    private String audioFilePath;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_audio_recorder);

        Button startButton = findViewById(R.id.startButton);
        Button stopButton = findViewById(R.id.stopButton);

        audioFilePath = getExternalFilesDir(null).getAbsolutePath() + "/audio_record.aac";

        mediaRecorder = new MediaRecorder();
        mediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
        mediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP);
        mediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
        mediaRecorder.setOutputFile(audioFilePath);

        startButton.setOnClickListener(v -> {
            Log.i("wtf", "startRecording");
            startRecording();
        });

        stopButton.setOnClickListener(v -> {
            Log.i("wtf", "stopRecording");
            stopRecording();
        });
    }
}

Start recording:

Java
    public void startRecording() {
        try {
            mediaRecorder.prepare();
            mediaRecorder.start();
        } catch (IOException e) {
            System.out.print("Startup exception");
            e.printStackTrace();
        }
    }

Stop recording:

Java
    public void stopRecording() {
        try {
            mediaRecorder.stop();
            mediaRecorder.release();
        } catch (RuntimeException e) {
            e.printStackTrace();
        }
    }

Release resources when the page is destroyed:

Java
    @Override
    protected void onDestroy() {
        super.onDestroy();
        if (mediaRecorder != null) {
            mediaRecorder.release();
        }
    }
}

Recording can also be started and stopped through buttons or other controls:

Java
Button startButton = findViewById(R.id.startButton);
Button stopButton = findViewById(R.id.stopButton);

startButton.setOnClickListener(v -> startRecording());
stopButton.setOnClickListener(v -> stopRecording());

7.2 Capturing PCM with AudioRecord​

AudioRecord is a lower-level audio capture interface and is suitable for low-latency recording, real-time audio processing, and custom audio algorithms.

Add the recording permission:

XML
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-feature android:name="android.hardware.microphone" />

Initialize AudioRecord:

Java
public class AudioRecorderActivity extends AppCompatActivity {
    private static final int SAMPLE_RATE_IN_HZ = 44100;
    private AudioRecord audioRecord;
    private boolean isRecording = false;
    private Thread recordingThread;
    private String audioFilePath;
    private FileOutputStream fileOutputStream;

    @Override
    @RequiresPermission(Manifest.permission.RECORD_AUDIO)
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_audio_recorder);

        Button startButton = findViewById(R.id.startButton);
        Button stopButton = findViewById(R.id.stopButton);

        int bufferSize = AudioRecord.getMinBufferSize(
                SAMPLE_RATE_IN_HZ,
                AudioFormat.CHANNEL_IN_MONO,
                AudioFormat.ENCODING_PCM_16BIT
        );

        audioRecord = new AudioRecord(
                MediaRecorder.AudioSource.MIC,
                SAMPLE_RATE_IN_HZ,
                AudioFormat.CHANNEL_IN_MONO,
                AudioFormat.ENCODING_PCM_16BIT,
                bufferSize
        );

        recordingThread = new Thread(new Runnable() {
            @Override
            public void run() {
                recordAudio();
            }
        });

        startButton.setOnClickListener(v -> {
            Log.i("wtf", "startRecording");
            startRecording();
        });

        stopButton.setOnClickListener(v -> {
            Log.i("wtf", "stopRecording");
            stopRecording();
        });
    }
}

Start recording:

Java
    public void startRecording() {
        audioRecord.startRecording();
        isRecording = true;
        recordingThread.start();
    }

Stop recording:

Java
    public void stopRecording() {
        isRecording = false;
        audioRecord.stop();
        audioRecord.release();
    }

Read audio data and save it as a WAV file:

Java
    private void recordAudio() {
        String audioFileName = "audio_record_" + System.currentTimeMillis() + ".wav";
        File audioFile = new File(getExternalFilesDir(null), audioFileName);
        audioFilePath = audioFile.getAbsolutePath();

        byte[] audioBuffer = new byte[1024];
        int totalBytesRead = 0;

        try (FileOutputStream fos = new FileOutputStream(audioFile)) {
            writeWavHeader(fos, 0, 0);

            while (isRecording) {
                int read = audioRecord.read(audioBuffer, 0, audioBuffer.length);
                if (read > 0) {
                    fos.write(audioBuffer, 0, read);
                    totalBytesRead += read;
                }
            }

            updateWavHeader(audioFile, totalBytesRead);

        } catch (IOException e) {
            Log.e("AudioRecord", "File write failed: " + e.getMessage());
        }
    }

Write the WAV file header:

Java
    private void writeWavHeader(FileOutputStream out, long totalAudioLen, long totalDataLen) throws IOException {
        long sampleRate = SAMPLE_RATE_IN_HZ;
        int channels = 1;
        int bitsPerSample = 16;

        byte[] header = new byte[44];

        header[0] = 'R'; header[1] = 'I'; header[2] = 'F'; header[3] = 'F';
        header[4] = (byte) (totalDataLen & 0xff);
        header[5] = (byte) ((totalDataLen >> 8) & 0xff);
        header[6] = (byte) ((totalDataLen >> 16) & 0xff);
        header[7] = (byte) ((totalDataLen >> 24) & 0xff);

        header[8] = 'W'; header[9] = 'A'; header[10] = 'V'; header[11] = 'E';

        header[12] = 'f'; header[13] = 'm'; header[14] = 't'; header[15] = ' ';

        header[16] = 16; header[17] = 0; header[18] = 0; header[19] = 0;

        header[20] = 1; header[21] = 0;

        header[22] = (byte) channels; header[23] = 0;

        header[24] = (byte) (sampleRate & 0xff);
        header[25] = (byte) ((sampleRate >> 8) & 0xff);
        header[26] = (byte) ((sampleRate >> 16) & 0xff);
        header[27] = (byte) ((sampleRate >> 24) & 0xff);

        long byteRate = sampleRate * channels * bitsPerSample / 8;
        header[28] = (byte) (byteRate & 0xff);
        header[29] = (byte) ((byteRate >> 8) & 0xff);
        header[30] = (byte) ((byteRate >> 16) & 0xff);
        header[31] = (byte) ((byteRate >> 24) & 0xff);

        header[32] = (byte) (channels * bitsPerSample / 8);
        header[33] = 0;

        header[34] = (byte) bitsPerSample;
        header[35] = 0;

        header[36] = 'd'; header[37] = 'a'; header[38] = 't'; header[39] = 'a';
        header[40] = (byte) (totalAudioLen & 0xff);
        header[41] = (byte) ((totalAudioLen >> 8) & 0xff);
        header[42] = (byte) ((totalAudioLen >> 16) & 0xff);
        header[43] = (byte) ((totalAudioLen >> 24) & 0xff);

        out.write(header, 0, 44);
    }

Update the data size in the WAV file header:

Java
    private void updateWavHeader(File file, int totalAudioLen) {
        try (RandomAccessFile raf = new RandomAccessFile(file, "rw")) {
            long totalDataLen = totalAudioLen + 36;
            raf.seek(4);
            raf.write((int) (totalDataLen & 0xff));
            raf.write((int) ((totalDataLen >> 8) & 0xff));
            raf.write((int) ((totalDataLen >> 16) & 0xff));
            raf.write((int) ((totalDataLen >> 24) & 0xff));

            raf.seek(40);
            raf.write((int) (totalAudioLen & 0xff));
            raf.write((int) ((totalAudioLen >> 8) & 0xff));
            raf.write((int) ((totalAudioLen >> 16) & 0xff));
            raf.write((int) ((totalAudioLen >> 24) & 0xff));
        } catch (IOException e) {
            Log.e("AudioRecord", "Error updating WAV header: " + e.getMessage());
        }
    }

Release resources when the page is destroyed:

Java
    @Override
    protected void onDestroy() {
        super.onDestroy();
        if (audioRecord != null) {
            audioRecord.release();
        }
    }
}

Recording can also be started and stopped through buttons or other controls:

Java
Button startButton = findViewById(R.id.startButton);
Button stopButton = findViewById(R.id.stopButton);

startButton.setOnClickListener(v -> startRecording());
stopButton.setOnClickListener(v -> stopRecording());

To save recording data, write the audio stream to a file through AudioRecord, or use MediaRecorder directly to save a .3gp file.

7.3 Notes​

  1. Permission: confirm in system settings that microphone permission has been granted to avoid runtime permission prompts.
  2. Device compatibility: make sure the device supports microphone functionality, and check the related capabilities of AudioManager and MediaRecorder.
  3. Thread management: when using AudioRecord, read audio data in a background thread to avoid ANR.
  4. Use MediaRecorder for simple recording.
  5. Use AudioRecord when finer control is required, such as real-time audio processing.

8. Speaker Usage​

The speaker can play local audio files through Android audio APIs. Common options include:

APIDescription
MediaPlayerPlays common audio files.
AudioTrackPlays PCM data.

Before playback, verify that the media volume, audio file path, and file format are correct.

The following example uses MediaPlayer to play the system default notification sound, which can be used to quickly verify whether the speaker outputs audio correctly.

Kotlin
package com.example.head

import android.app.Activity
import android.media.MediaPlayer
import android.media.RingtoneManager
import android.os.Bundle
import android.widget.Button

class SpeakerPlayActivity : Activity() {
    private var mediaPlayer: MediaPlayer? = null

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        val playButton = Button(this).apply { text = "Play system audio" }
        setContentView(playButton)
        playButton.setOnClickListener { playSystemSound() }
    }

    private fun playSystemSound() {
        mediaPlayer?.release()
        val uri = RingtoneManager.getDefaultUri(RingtoneManager.TYPE_NOTIFICATION)
        mediaPlayer = MediaPlayer.create(this, uri).apply {
            setOnCompletionListener {
                it.release()
                mediaPlayer = null
            }
            start()
        }
    }

    override fun onDestroy() {
        mediaPlayer?.release()
        mediaPlayer = null
        super.onDestroy()
    }
}

9. SPI Eye Screen Control​

The SPI eye screen supports smile, star eyes, speechless expression, closed eyes, blinking, eye position control, and other functions.

9.1 Integration​

To integrate the SPI eye screen control library, place libSpiScreen.aar in the libs directory of the application module. Then add the following dependency in build.gradle:

Gradle
dependencies {
    implementation(files("./libs/libSpiScreen.aar"))
}

9.2 Eye Control Example​

Kotlin

EyeAction.openSpiEye()
// Eye control methods
EyeAction.blinkEye() // Blink once
EyeAction.blinkEye(240, 2) // Blink twice
EyeAction.updateEmotionSmile() // Smile
EyeAction.closeEye()  // Close eyes
EyeAction.updateEmotionWuYu()   // Speechless expression
EyeAction.updateEmotionStar()  // Star eyes

// Control eye position. x: -100 to 100, y: -100 to 100. Center point is 0. Positive y is downward, and positive x is to the right.
// Control both eye positions at the same time
EyeAction.updateBothScleraXY( e0.x, e0.y, e1.x, e1.y )
// Control one eye position. 0 indicates the left eye, and 1 indicates the right eye.
EyeAction.updateScleraXY(0, e0.x, e0.y)

// Close the SPI screen
EyeAction.closeSpiEye()

10. Notes​

  • Camera and microphone permissions must be granted before use.
  • The camera, microphone, and speaker may be occupied by other applications. If debugging issues occur, check whether these resources are already in use.
  • Release resources at the end of the lifecycle after using the camera, audio recording, or eye screen.
  • AudioRecord must read audio data in a background thread.
  • Open the SPI eye screen before use, and close it when the application exits or the screen is no longer needed.