LangChain Python SDK
You can use LangChain Python SDK to interact with FriendliAI. This makes migration of existing applications already using LangChain particularly easy.
How to use
Before you start, ensure you’ve already obtained the FRIENDLI_TOKEN
from the Friendli Suite.
Our products are entirely compatible with OpenAI, so we use the langchain-openai
package by referring to the FriendliAI baseURL
.
pip install -qU langchain-openai langchain
Instantiation
Now we can instantiate our model object and generate chat completions. We provide usage examples for each type of endpoint. Choose the one that best suits your needs:
Runnable interface
We support both synchronous and asynchronous runnable methods to generate a response.
Synchronous methods:
Asynchronous methods:
Chaining
We can chain our model with a prompt template. Prompt templates convert raw user input to better input to the LLM.
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_messages([
("system", "You are a world class technical documentation writer."),
("user", "{input}")
])
chain = prompt | llm
print(chain.invoke({"input": "how can langsmith help with testing?"}))
To get the string value instead of the message, we can add an output parser to the chain.
from langchain_core.output_parsers import StrOutputParser
output_parser = StrOutputParser()
chain = prompt | llm | output_parser
print(chain.invoke({"input": "how can langsmith help with testing?"}))
Tool calling
Describe tools and their parameters, and let the model return a tool to invoke with the input arguments. Tool calling is extremely useful for enhancing the model’s capability to provide more comprehensive and actionable responses.
Define tools to use
The @tool
decorator is used to define a tool.
If you set parse_docstring=True
, the tool will parse the docstring to extract the information of arguments.
Bind tools to the model
Now models can generate a tool calling response.
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="meta-llama-3.1-8b-instruct",
base_url="https://inference.friendli.ai/v1",
api_key=os.environ["FRIENDLI_TOKEN"],
)
llm_with_tools = llm.bind_tools(tools)
query = "What is 3 * 12? Also, what is 11 + 49?"
print(llm_with_tools.invoke(query).tool_calls)
Generate a tool assisted message
Use the tool call results to generate a message.
from langchain_core.messages import HumanMessage, ToolMessage
messages = [HumanMessage(query)]
ai_msg = llm_with_tools.invoke(messages)
messages.append(ai_msg)
for tool_call in ai_msg.tool_calls:
selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
tool_output = selected_tool.invoke(tool_call["args"])
messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))
print(llm_with_tools.invoke(messages))
For more information on how to use tools, check out the LangChain documentation.