Skip to content

Commit b609a73

Browse files
PaulAsjesPaulAsjes
authored
Merge pull request #54 from elevenlabs/louis/rtc-fixes
louis/rtc fixes
2 parents 7d471e9 + 0628818 commit b609a73

File tree

10 files changed

+819
-296
lines changed

10 files changed

+819
-296
lines changed

‎Package.resolved‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Package.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ let package = Package(
1515
],
1616
dependencies: [
1717
.package(url: "https://github.com/livekit/client-sdk-swift.git", from: "2.0.0"),
18-
.package(url: "https://github.com/devicekit/DeviceKit.git", from: "5.8.0"),
18+
.package(url: "https://github.com/devicekit/DeviceKit.git", from: "5.6.0"),
1919
],
2020
targets: [
2121
.target(

‎README.md‎

Lines changed: 215 additions & 161 deletions
Original file line numberDiff line numberDiff line change
@@ -1,161 +1,215 @@
1-
# Elevenlabs Conversational AI Swift SDK
2-
3-
![convai222](https://github.com/user-attachments/assets/ca4fa726-5e98-4bbc-91b2-d055e957df7d)
4-
5-
Elevenlabs Conversational AI Swift SDK is a framework designed to integrate ElevenLabs' powerful conversational AI capabilities into your Swift applications. Leverage advanced audio processing and seamless WebSocket communication to create interactive and intelligent conversational voice experiences.
6-
7-
For detailed documentation, visit the [ElevenLabs Swift SDK documentation](https://elevenlabs.io/docs/conversational-ai/libraries/conversational-ai-sdk-swift).
8-
9-
> [!NOTE]
10-
> This library is launching to primarily support Conversational AI. The support for speech synthesis and other more generic use cases is planned for the future.
11-
12-
## Install
13-
14-
Add the Elevenlabs Conversational AI Swift SDK to your project using Swift Package Manager:
15-
16-
1. Open Your Project in Xcode
17-
- Navigate to your project directory and open it in Xcode.
18-
2. Add Package Dependency
19-
- Go to `File` > `Add Packages...`
20-
3. Enter Repository URL
21-
- Input the following URL: `https://github.com/elevenlabs/ElevenLabsSwift`
22-
4. Select Version
23-
5. Import the SDK
24-
```swift
25-
import ElevenLabsSDK
26-
```
27-
6. Ensure `NSMicrophoneUsageDescription` is added to your Info.plist to explain microphone access.
28-
29-
## Usage
30-
31-
### Setting Up a Conversation Session
32-
33-
1. Configure the Session
34-
Create a `SessionConfig` with either an `agentId` or `signedUrl`.
35-
36-
```swift
37-
let config = ElevenLabsSDK.SessionConfig(agentId: "your-agent-id")
38-
```
39-
40-
2. Define Callbacks
41-
Implement callbacks to handle various conversation events.
42-
43-
```swift
44-
var callbacks = ElevenLabsSDK.Callbacks()
45-
callbacks.onConnect = { conversationId in
46-
print("Connected with ID: \(conversationId)")
47-
}
48-
callbacks.onMessage = { message, role in
49-
print("\(role.rawValue): \(message)")
50-
}
51-
callbacks.onError = { error, info in
52-
print("Error: \(error), Info: \(String(describing: info))")
53-
}
54-
callbacks.onStatusChange = { status in
55-
print("Status changed to: \(status.rawValue)")
56-
}
57-
callbacks.onModeChange = { mode in
58-
print("Mode changed to: \(mode.rawValue)")
59-
}
60-
callbacks.onVolumeUpdate = { volume in
61-
print("Input volume: \(volume)")
62-
}
63-
callbacks.onOutputVolumeUpdate = { volume in
64-
print("Output volume: \(volume)")
65-
}
66-
callbacks.onMessageCorrection = { original, corrected, role in
67-
print("Message corrected - Original: \(original), Corrected: \(corrected), Role: \(role.rawValue)")
68-
}
69-
```
70-
71-
3. Start the Conversation
72-
Initiate the conversation session asynchronously.
73-
74-
```swift
75-
Task {
76-
do {
77-
let conversation = try await ElevenLabsSDK.startSession(config: config, callbacks: callbacks)
78-
// Use the conversation instance as needed
79-
} catch {
80-
print("Failed to start conversation: \(error)")
81-
}
82-
}
83-
```
84-
85-
### Advanced Configuration
86-
87-
1. Using Client Tools
88-
89-
```swift
90-
var clientTools = ElevenLabsSDK.ClientTools()
91-
clientTools.register("weather") { parameters in
92-
print("Weather parameters received:", parameters)
93-
// Handle the weather tool call and return response
94-
return "The weather is sunny today"
95-
}
96-
97-
let conversation = try await ElevenLabsSDK.startSession(
98-
config: config,
99-
callbacks: callbacks,
100-
clientTools: clientTools
101-
)
102-
```
103-
104-
2. Using Overrides
105-
106-
```swift
107-
let overrides = ElevenLabsSDK.ConversationConfigOverride(
108-
agent: ElevenLabsSDK.AgentConfig(
109-
prompt: ElevenLabsSDK.AgentPrompt(prompt: "You are a helpful assistant"),
110-
language: .en
111-
)
112-
)
113-
114-
let config = ElevenLabsSDK.SessionConfig(
115-
agentId: "your-agent-id",
116-
overrides: overrides
117-
)
118-
```
119-
120-
### Managing the Session
121-
122-
- End Session
123-
124-
```swift
125-
conversation.endSession()
126-
```
127-
128-
- Control Recording
129-
130-
```swift
131-
conversation.startRecording()
132-
conversation.stopRecording()
133-
```
134-
135-
- Send Messages and Updates
136-
137-
```swift
138-
// Send a contextual update to the conversation
139-
conversation.sendContextualUpdate("The user is now in the kitchen")
140-
141-
// Send a user message directly
142-
conversation.sendUserMessage("Hello, how are you?")
143-
144-
// Send user activity signal
145-
conversation.sendUserActivity()
146-
```
147-
148-
- Volume Controls
149-
150-
```swift
151-
// Get current input/output volume levels
152-
let inputVolume = conversation.getInputVolume()
153-
let outputVolume = conversation.getOutputVolume()
154-
155-
// Set conversation volume (0.0 to 1.0)
156-
conversation.conversationVolume = 0.8
157-
```
158-
159-
## Example
160-
161-
Explore examples in the [ElevenLabs Examples repository](https://github.com/elevenlabs/elevenlabs-examples/tree/main/examples/conversational-ai/swift).
1+
# ElevenLabs Conversational AI Swift SDK
2+
3+
<img src="https://github.com/user-attachments/assets/ca4fa726-5e98-4bbc-91b2-d055e957df7d" alt="ElevenLabs ConvAI" width="400">
4+
5+
A Swift SDK for integrating ElevenLabs' conversational AI capabilities into your iOS and macOS applications. Built on top of LiveKit WebRTC for real-time audio streaming and communication.
6+
7+
[![Swift Package Manager](https://img.shields.io/badge/Swift%20Package%20Manager-compatible-brightgreen.svg)](https://github.com/apple/swift-package-manager)
8+
[![Platform](https://img.shields.io/badge/platform-iOS%20%7C%20macOS-lightgrey.svg)](https://github.com/elevenlabs/ElevenLabsSwift)
9+
10+
## Quick Start
11+
12+
### Installation
13+
14+
Add to your project using Swift Package Manager:
15+
16+
```swift
17+
dependencies: [
18+
.package(url: "https://github.com/elevenlabs/ElevenLabsSwift.git", from: "1.2.0")
19+
]
20+
```
21+
22+
### Basic Usage
23+
24+
```swift
25+
import ElevenLabsSwift
26+
27+
// 1. Configure your session
28+
let config = ElevenLabsSDK.SessionConfig(agentId: "your-agent-id")
29+
30+
// 2. Set up callbacks
31+
var callbacks = ElevenLabsSDK.Callbacks()
32+
callbacks.onConnect = { conversationId in
33+
print("🟢 Connected: \(conversationId)")
34+
}
35+
callbacks.onMessage = { message, role in
36+
print("💬 \(role.rawValue): \(message)")
37+
}
38+
callbacks.onError = { error, _ in
39+
print("❌ Error: \(error)")
40+
}
41+
42+
// 3. Start conversation
43+
Task {
44+
do {
45+
let conversation = try await ElevenLabsSDK.startSession(
46+
config: config,
47+
callbacks: callbacks
48+
)
49+
50+
// Send messages
51+
conversation.sendUserMessage("Hello!")
52+
conversation.sendContextualUpdate("User is in the kitchen")
53+
54+
// Control recording
55+
conversation.startRecording()
56+
conversation.stopRecording()
57+
58+
// End session
59+
conversation.endSession()
60+
} catch {
61+
print("Failed to start conversation: \(error)")
62+
}
63+
}
64+
```
65+
66+
### Requirements
67+
68+
- iOS 16.0+ / macOS 10.15+
69+
- Swift 5.9+
70+
- Add `NSMicrophoneUsageDescription` to your Info.plist
71+
72+
## Advanced Features
73+
74+
### Private agents
75+
76+
For private agents that require authentication, provide a conversation token in your `SessionConfig`.
77+
78+
The conversation token should be generated on your backend with a valid ElevenLabs API key. Do NOT store the API key within your app.
79+
80+
```js
81+
// Node.js server
82+
app.get("/api/conversation-token", yourAuthMiddleware, async (req, res) => {
83+
const response = await fetch(
84+
`https://api.elevenlabs.io/v1/convai/conversation/token?agent_id=${process.env.AGENT_ID}`,
85+
{
86+
headers: {
87+
// Requesting a conversation token requires your ElevenLabs API key
88+
// Do NOT expose your API key to the client!
89+
'xi-api-key': process.env.ELEVENLABS_API_KEY,
90+
}
91+
}
92+
);
93+
94+
if (!response.ok) {
95+
return res.status(500).send("Failed to get conversation token");
96+
}
97+
98+
const body = await response.json();
99+
res.send(body.token);
100+
);
101+
```
102+
103+
```swift
104+
guard let url = URL(string: "https://your-backend-api.com/api/conversation-token") else {
105+
throw URLError(.badURL)
106+
}
107+
108+
// Create request. This is a simple implementation, in a real world app you should add security headers
109+
var request = URLRequest(url: url)
110+
request.httpMethod = "GET"
111+
112+
// Make request
113+
let (data, _) = try await URLSession.shared.data(for: request)
114+
115+
// Parse response
116+
let response = try JSONDecoder().decode([String: String].self, from: data)
117+
guard let conversationToken = response["conversationToken"] else {
118+
throw NSError(domain: "TokenError", code: 0, userInfo: [NSLocalizedDescriptionKey: "No token received"])
119+
}
120+
121+
// Agent ID isn't required when providing a conversation token
122+
let config = ElevenLabsSDK.SessionConfig(conversationToken: conversationToken)
123+
124+
let conversation = try await ElevenLabsSDK.startSession(config: config)
125+
```
126+
127+
### Client Tools
128+
129+
Register custom tools that your agent can call:
130+
131+
```swift
132+
var clientTools = ElevenLabsSDK.ClientTools()
133+
clientTools.register("get_weather") { parameters in
134+
let location = parameters["location"] as? String ?? "Unknown"
135+
return "The weather in \(location) is sunny, 72°F"
136+
}
137+
138+
let conversation = try await ElevenLabsSDK.startSession(
139+
config: config,
140+
callbacks: callbacks,
141+
clientTools: clientTools
142+
)
143+
```
144+
145+
### Agent Configuration
146+
147+
Override agent settings:
148+
149+
```swift
150+
let overrides = ElevenLabsSDK.ConversationConfigOverride(
151+
agent: ElevenLabsSDK.AgentConfig(
152+
prompt: ElevenLabsSDK.AgentPrompt(prompt: "You are a helpful cooking assistant"),
153+
language: .en,
154+
firstMessage: "Hello! How can I help you cook today?"
155+
),
156+
tts: ElevenLabsSDK.TTSConfig(voiceId: "your-voice-id")
157+
)
158+
159+
let config = ElevenLabsSDK.SessionConfig(
160+
agentId: "your-agent-id",
161+
overrides: overrides
162+
)
163+
```
164+
165+
### Audio Controls
166+
167+
```swift
168+
// Volume management
169+
conversation.conversationVolume = 0.8
170+
let inputLevel = conversation.getInputVolume()
171+
let outputLevel = conversation.getOutputVolume()
172+
173+
// Recording controls
174+
conversation.startRecording()
175+
conversation.stopRecording()
176+
177+
// Volume callbacks
178+
callbacks.onVolumeUpdate = { level in
179+
print("🎤 Input: \(level)")
180+
}
181+
callbacks.onOutputVolumeUpdate = { level in
182+
print("🔊 Output: \(level)")
183+
}
184+
```
185+
186+
## Architecture
187+
188+
The SDK is built with clean architecture principles:
189+
190+
```
191+
ElevenLabsSDK (Main API)
192+
├── LiveKitConversation (WebRTC Management)
193+
├── RTCLiveKitAudioManager (Audio Streaming)
194+
├── DataChannelManager (Message Handling)
195+
└── NetworkService (Token Management)
196+
```
197+
198+
## Examples
199+
200+
Check out complete examples in the [ElevenLabs Examples repository](https://github.com/elevenlabs/elevenlabs-examples/tree/main/examples/conversational-ai/swift).
201+
202+
## Contributing
203+
204+
We welcome contributions! Please check out our [Contributing Guide](CONTRIBUTING.md) and join us in the [ElevenLabs Discord](https://discord.gg/elevenlabs).
205+
206+
## License
207+
208+
This SDK is licensed under the MIT License. See [LICENSE](LICENSE) for details.
209+
210+
## Support
211+
212+
- 📚 [Documentation](https://elevenlabs.io/docs/conversational-ai/libraries/conversational-ai-sdk-swift)
213+
- 💬 [Discord Community](https://discord.gg/elevenlabs)
214+
- 🐛 [Issues](https://github.com/elevenlabs/ElevenLabsSwift/issues)
215+
- 📧 [Support Email](mailto:[email protected])

0 commit comments

Comments
 (0)