SparkOS Harmony NEXT接入文档

# SparkOS Harmony NEXT接入文档

## 1. 兼容性说明

|类别|兼容范围|
|---|---|
|系统|SDK支持HarmonyOS 5.0.0(SDK API Level 12)及以上系统版本，支持arm64-v8a架构|
|开发语言|ArkTS|
## 2. 准备工作

### 2.1 平台配置

在集成开发SDK前，需要开发者在AIUI交互平台上创建应用及申请授权，设置好相关的配置，如个性化数据、技能、情景以及自然语言理解(即语义是使用交互大模型还是AIUI)等，并获取AIUI交互协议文档。

### 2.2 授权申请

接入SDK前，请先联系项目侧开通使用交互大模型需要用到的各种云端授权，防止出现运行时的授权报错。

除了云端功能需要授权外，如果需要使用到多模交互、音频编解码等功能，也需要额外再开通客户端侧的离线能力授权，具体可以联系SDK技术支持人工协助。

### 2.3 功能介绍

SDK内部包含多个独立子能力，通过编排协议按业务流程编排形成AIUI交互流程，预置子能力如下：

|子能力|说明|
|---|---|
|录音器、播放器|用于音频采集和播放|
|视频录像|用于采集视频/图片|
|VAD/IAT/NLP/TTS|在线AIUI能力，进行人声检测/语音识别/语义理解/合成|
|AudioCodec|音频编解码，用于将采集的音频编码、服务下发的音频解码(需要开通授权)|
|ImageConvert|图片编码，用于将采集的视频/图片编码(需要开通授权)|
|虚拟形象|AI虚拟人技术，用于将合成结果通过虚拟形象播报(需要开通授权)|
## 3. SDK组成

SDK zip包解压后文件结构：

```Plain Text

├── SparkOS Harmony NEXT接入文档.pdf // AIUI SDK能力集成指南
├── AIUI_APPDemo // SDK的使用Demo，集成前通过Demo了解调用原理
├── ReleaseNote.txt // SDK版本日志
├── AIKit.har // 核心框架依赖
└── XRTCSDK_{version}.har // 虚拟人视频拉流依赖组件
```

## 4. SDK工程配置

### 4.1 导入SDK库

将AIKit.har文件复制到项目模块目录(默认entry)->libs(无则手动创建)，并在模块下的oh-package.json5中添加依赖：

```JSON

{
  "name": "entry",
  "version": "1.0.0",
  "description": "Please describe the basic information.",
  "main": "",
  "author": "",
  "license": "",
  "dependencies": {
    "aikit": "file:./libs/AIKit.har"
  }
}
```

### 4.2 配置权限

在模块module.json5(模块名称/src/main/)中声明网络和设备权限：

```JSON

{
  "module": {
    "name": "entry",
    "type": "entry",
    "description": "$string:module_desc",
    "mainElement": "EntryAbility",
    "deviceTypes": [
      "phone",
      "tablet",
      "2in1"
    ],
    "requestPermissions": [
      {
        "name": "ohos.permission.APP_TRACKING_CONSENT",
        "reason": "$string:MainAbility_desc",
        "usedScene": {
          "abilities": [
            "EntryFormAbility"
          ],
          "when": "inuse"
        }
      },
      {
        "name": "ohos.permission.STORE_PERSISTENT_DATA",
        "reason": "$string:MainAbility_desc",
        "usedScene": {
          "abilities": [
            "MainAbility"
          ],
          "when": "always"
        }
      },
      {
        "name": "ohos.permission.INTERNET",
        "reason": "$string:MainAbility_desc",
        "usedScene": {
          "abilities": [
            "MainAbility"
          ],
          "when": "always"
        }
      },
      {
        "name": "ohos.permission.MICROPHONE",
        "reason": "$string:MainAbility_desc",
        "usedScene": {
          "abilities": [
            "MainAbility"
          ],
          "when": "always"
        }
      },
      {
        "name": "ohos.permission.CAMERA",
        "reason": "$string:MainAbility_desc",
        "usedScene": {
          "abilities": [
            "MainAbility"
          ],
          "when": "always"
        }
      },
      {
        "name": "ohos.permission.APPROXIMATELY_LOCATION",
        "reason": "$string:MainAbility_desc",
        "usedScene": {
          "abilities": [
            "MainAbility"
          ],
          "when": "always"
        }
      }
    ]
  }
}
```

## 5. 接口调用流程

### 5.1 初始化

使用AIUI能力前，需获取授权应用信息(appId、apiKey、apiSecret)，联系项目经理获取；SDK仅需初始化一次。

#### 初始化示例代码

```TypeScript

// 创建一个类来实现AuthListener接口
class MyAuthListener implements AuthListener {
  // 实现接口中的方法
  onAuthStateChange(type: number, code: number): void {
    hilog.info(0x0000, 'AEE', 'onAuthStateChange:%{public}d,%{public}d', code, type);
  }
}
const context = getContext(this);

//初始化参数，填入您的appID、apiKey、apiSecret
let initParams: InitParams = InitParams.builder()
  .appID('您的appID')
  .apiKey('您的apiKey')
  .apiSecret('您的apiSecret')
  .workDir(context.filesDir) //SDK运行目录，需要有写权限
  .authType(0) //授权类型 0:设备授权 1:应用授权
  .logLevel(LogLvl.VERBOSE) //日志级别
  .logMode(2) //Native日志模式 1:控制台打印 2:写入文件
  .logPath(workDir + '/AIUILog_' + Utils.getCurrentDate() + '.txt') //Native日志路径
  .build();

//注册能力结果回调(可选)
aiListener: AiResponseListener = {
  onResult: (ability: string, handleID: number, responseData: AiResponse[], usrContext: object | null): void => {
    //能力结果回调
  },
  onEvent: (ability: string, handleID: number, event: number, eventData: AiResponse[], usrContext: object | null): void => {
    //能力进度回调
  },
  onError: (ability: string, handleID: number, err: number, errMsg: string, usrContext: object | null): void => {
    //能力错误回调
  }
}
AppHelper.getInst().registerListener(this.aiListener)

//初始化
AppHelper.getInst().init(context, initParams, authListener, aiListener)
```

#### 初始化参数说明

|参数|类型|是否必填|说明|
|---|---|---|---|
|appID|string|是|AIUI交互平台创建应用后生成的应用ID|
|apiKey|string|是|AIUI交互平台创建应用后生成的唯一应用标识|
|apiSecret|string|是|AIUI交互平台创建应用后生成的唯一应用秘钥|
|workDir|string|是|SDK工作目录，用户可自行指定，需确保有访问权限|
|authType|number|否|端能力授权方式，0=设备级授权，1=应用级授权|
|logLevel|LogLvl|否|日志输出等级：DEBUG/WARN/INFO/ERROR/FATAL|
|logMode|number|否|模式：stdout:0 控制台:1 file:2|
|logPath|string|否|输出模式为文件时的文件路径+名称|
#### 能力回调接口

```TypeScript

export interface AiResponseListener {
  onResult(ability: string, handleID: number, responseData: AiResponse[], usrContext: object | null): void;
  onEvent(ability: string, handleID: number, event: number, eventData: AiResponse[], usrContext: object | null): void;
  onError(ability: string, handleID: number, err: number, msg: string, usrContext: object | null): void;
}
```

### 5.2 请求调用

#### 5.2.1 场景切换

通过`AppHelper.start`启动会话，`AppHelper.stop`结束会话，支持4类场景：

- 语音输入，语音输出

- 语音输入，视频输出（虚拟人）

- 视频输入，语音输出（多模）

- 视频输入，视频输出（虚拟人+多模）

```TypeScript

// 应用模式定义
export enum AppMode {
  AUDIO_TO_AUDIO = 0,
  AUDIO_TO_VIDEO,
  VIDEO_TO_AUDIO,
  VIDEO_TO_VIDEO,
}

/**
* 启动会话
* @param mode 应用模式
* @param appParams 会话参数
* @return 错误码，0为成功
*/
start(mode: AppMode, appParams: AppParams): number

/**
* 停止会话
* @return 错误码，0为成功
*/
stop(): number
```

#### 5.2.2 参数配置

```TypeScript

export class AppParams {}
// 建造者类
export class AppParamsBuilder {
  isDuplex(isDuplex: boolean): AppParamsBuilder
  uid(uid: string): AppParamsBuilder
  scene(scene: string): AppParamsBuilder
  prompt(prompt: string): AppParamsBuilder
  chatId(chatId: string): AppParamsBuilder
  eos(eosDelay: number): AppParamsBuilder
  avatarSize(width: number, height: number): AppParamsBuilder
  avatarId(avatarId: string): AppParamsBuilder
  vcn(vcn: string): AppParamsBuilder
  imageSrcFormat(srcFormat: string): AppParamsBuilder
  imageDetectBlurry(detectBlurry: boolean): AppParamsBuilder
  imageSaveResult(imageSaveResult: boolean): AppParamsBuilder
  imageSaveResultPath(saveResultPath: string): AppParamsBuilder
  imageRotation(rotation: number): AppParamsBuilder
  latitude(latitude: number): AppParamsBuilder
  longitude(longitude: number): AppParamsBuilder
  surfaceId(surfaceId: string): AppParamsBuilder
  build(): AppParams
}
```

#### start传参完整示例

```TypeScript

public start(scene: string, eos: number, vcn: string, isOnDuplex:boolean, enableVms: boolean, avatarId: string, avatarImageUrl: string, enableCamera: boolean, surfaceId?: string, imageSaveResult?: boolean): number {
  //构建AIUI参数
  let paramBuilder = AppParams.builder()
  paramBuilder.uid('HarmonyOS_AIUI_TEST') //用户唯一标识
    .scene(scene) //网页后台配置的场景
    .eos(eos) //vad后端点
    .vcn(vcn) //发音人
    .chatId('XXXXXXXX') //会话id
  //虚拟人
  if (enableVms) {
    paramBuilder.avatarSize(512, 512) //虚拟人尺寸
      .avatarId(avatarId) //虚拟人形象ID
    if (!AIKitUtil.isStrEmpty(avatarImageUrl)) {
      paramBuilder.avatarImage(avatarImageUrl).avatarImageEncoding("jpg")
    }
  }
  if (enableCamera) {
    paramBuilder.imageSrcFormat("raw_nv21")
      .imageDetectBlurry(true)
    if (imageSaveResult) {
      paramBuilder.imageSaveResult(imageSaveResult)
      let filesDir = getContext(this).filesDir;
      AIKitUtil.checkAndCreateDirectory(filesDir + '/imageTransform')
      paramBuilder.imageSaveResultPath(filesDir + '/imageTransform')
    }
    paramBuilder.imageRotation(90)
    if(surfaceId){
      paramBuilder.surfaceId(surfaceId)
    }
  }
  paramBuilder.isDuplex(isOnDuplex);
  let mode = AppMode.AUDIO_TO_AUDIO
  if (enableVms) {
    mode = enableCamera ? AppMode.VIDEO_TO_VIDEO : AppMode.AUDIO_TO_VIDEO
  } else if (enableCamera) {
    mode = AppMode.VIDEO_TO_AUDIO
  }
  let ret = AppHelper.getInst().start(mode, paramBuilder.build())
  return ret;
}
```

#### 5.2.3 自定义IO

```TypeScript

export interface IDeviceIO {
  init(context: common.Context): void
  release(): void
}

export interface IDeviceVideoInput extends IDeviceIO {
  on(writer: IWriterWH): void
  off(): void
  switchTo(vds: VideoDataSource, surfaceId?: string): void
}

export interface IWriterWH {
  write(data: ArrayBuffer, width: number, height: number): void
}

export enum VideoDataSource {
  DEFAULT_CAMERA = 0,
  FRONT_CAMERA,
  REAR_CAMERA
}

export interface IDeviceOutput extends IDeviceIO {
  on(): void
  off(): void
}

export interface IDeviceVideoOutput extends IDeviceOutput{
  ready(): void
  pause(): void
  resume(): void
  data(data: ArrayBuffer, status?: AiStatus): void
  params(): VideoParams
  containerId(): string
}

export interface IDeviceAudioOutput extends IDeviceOutput{
  pause(): void
  resume(): void
  data(data: ArrayBuffer, status?: AiStatus): void
  isPlaying(): boolean
}

export interface IWriter {
  write(data: ArrayBuffer): void
}

export interface IDeviceAudioInput extends IDeviceIO{
  on(writer: IWriter): void
  off(): void
}
```

##### 注册自定义IO示例

- 注册播放器设备

```TypeScript

const deviceAudioOutput: IDeviceAudioOutput = {
  pause: (): void => { PcmStreamPlayer.getInst().pause() },
  resume: (): void => { PcmStreamPlayer.getInst().start(); },
  data: (data: ArrayBuffer, status?: AiStatus | undefined): void => {
    if (data.byteLength > 0) {
      PcmStreamPlayer.getInst().write(new Uint8Array(data));
    }
    if (status && status == AiStatus.END) {
      PcmStreamPlayer.getInst().onComplete()
    }
  },
  isPlaying: (): boolean => { return PcmStreamPlayer.getInst().isPlaying() },
  on: (): void => { PcmStreamPlayer.getInst().start(); },
  off: (): void => { PcmStreamPlayer.getInst().stop() },
  init: (context: Context): void => { PcmStreamPlayer.getInst().init() },
  release: (): void => { PcmStreamPlayer.getInst().release() }
}
AppHelper.getInst().registerAudioOutputDevice(deviceAudioOutput)
```

- 注册录音器设备

```TypeScript

const deviceAudioInput: IDeviceAudioInput = {
  on: (writer: IWriter): void => {
    const readDataCallback = (buffer: ArrayBuffer) => {
      if (writer) { writer.write(buffer) }
    };
    RecorderHelper.getInst().init(readDataCallback, audio.SourceType.SOURCE_TYPE_VOICE_COMMUNICATION)
    RecorderHelper.getInst().start()
  },
  off: (): void => {
    RecorderHelper.getInst().stop()
    RecorderHelper.getInst().release()
  },
  init: (context: Context): void => {},
  release: (): void => {}
}
AppHelper.getInst().registerAudioInputDevice(deviceAudioInput)
```

- 注册相机设备

```TypeScript

class CameraListener implements CameraHelperListener {
  writer: IWriterWH
  constructor(writer: IWriterWH) { this.writer = writer; }
  onImageAvailable(buffer: ArrayBuffer, width: number, height: number): void {
    this.writer.write(buffer, width, height);
  }
  onCameraOpened(width: number, height: number): void {}
}
const deviceVideoInput: IDeviceVideoInput = {
  on: (writer: IWriterWH): void => {
    let cameraListener: CameraListener = new CameraListener(writer)
    CameraHelper.getInst().setListener(cameraListener)
    CameraHelper.getInst().initCamera('相机控件的SurfaceId', this.mCameraPosition);
  },
  off: (): void => { CameraHelper.getInst().setListener(undefined) },
  switchTo: async (vds: VideoDataSource, surfaceId?: string | undefined): Promise<void> => {
    let cameraPosition = vds === VideoDataSource.FRONT_CAMERA ? camera.CameraPosition.CAMERA_POSITION_FRONT : camera.CameraPosition.CAMERA_POSITION_BACK
    await CameraHelper.getInst().initCamera('相机控件的SurfaceId', cameraPosition);
  },
  init: (context: Context): void => { CameraHelper.getInst().init(context) },
  release: (): void => { CameraHelper.getInst().releaseCamera() }
}
AppHelper.getInst().registerVideoInputDevice(deviceVideoInput)
```

- 注册视频输出设备

```TypeScript

const deviceVideoOutput: IDeviceVideoOutput = {
  ready: (): void => {},
  pause: (): void => {},
  resume: (): void => {},
  data: (data: ArrayBuffer): void => {},
  isPlaying: (): boolean => { return false },
  params: (): VideoParams => {
    let builder = VideoParams.builder()
      .rotation(VideoRotation.VIDEO_ROTATION_0)
      .fillMode(VideoFillMode.VIDEO_RENDER_MODE_FIT)
      .stream(VideoStreamType.VIDEO_STREAM_TYPE_BIG)
      .shape(VideoShape.VIDEO_SHAPE_ROUNDED)
      .viewId(AppConstants.DEFAULT_VMS_VIEWID)
    return builder.build()
  },
  containerId: (): string => { return AppConstants.DEFAULT_VMS_VIEWID },
  on: (): void => {},
  off: (): void => {},
  init: (context: Context): void => {},
  release: (): void => {}
};
AppHelper.getInst().registerVideoOutputDevice(deviceVideoOutput)
```

#### 5.2.4 交互协议

[https://yf2ljykclb.xfchat.iflytek.com/sync/Y98Ddchq6sGOfVbftySr8TuMzpb](https://yf2ljykclb.xfchat.iflytek.com/sync/Y98Ddchq6sGOfVbftySr8TuMzpb)

### 5.3 逆初始化

```TypeScript

AppHelper.getInst().release()
```

## 6. 高级功能

### 6.1 SDK初始化高级配置

|参数名|类型|说明|是否必填|
|---|---|---|---|
|authType|number|端能力授权方式，0=设备级授权，1=应用级授权|否|
|cfgPath|string|配置文件路径，需使用配置文件时传入|否|
### 6.2 动态配置访问url

```TypeScript

AiHelper.getInst().setConfig("HostUrl", '{"aiui_spark":"ws://sos-gray.cn-huabei-1.xf-yun.com/v3/sos","aiui_image":"ws://sos-gray.cn-huabei-1.xf-yun.com/v3/sos-image"}')
```

### 6.3 下发服务侧sid结果

```TypeScript

AiHelper.getInst().setConfig("ResultFormat", '{"aiui_spark":1}');
```

header数据格式：

```JSON

{
  "code": 0,
  "message": "success",
  "cid": "cidxxxxxxxxxxxxxxxxx",
  "sid": "xxx@xxxxxxxxxxxxxxxx"
}
```

### 6.4 获取会话结束标识

|key|status含义|获取方式|
|---|---|---|
|iat|0=第一帧，1=中间，2=最后一帧|onResult中key=iat取status|
|nlp|0=第一帧，1=中间，2=最后一帧|onResult中key=nlp取status|
|tts|0=第一帧，1=中间，2=最后一帧|onResult中key=tts取status|
|header|0=第一帧，1=中间，2=最后一帧|解析header的status字段|
### 6.5 日志配置

```TypeScript

setLogInfo(level: LogLvl, mode: number, path: string | null): number
```

|参数|类型|说明|是否必填|
|---|---|---|---|
|level|LogLvl|日志等级：VERBOSE/DEBUG/WARN/INFO/ERROR/FATAL|是|
|mode|number|模式：stdout:0 控制台:1 file:2|是|
|path|string|null|文件输出路径+名称|
## 7. 错误码

### 7.1 SDK错误码

参见AIKit错误码描述

### 7.2 云端错误码

参见云端错误码查询