LangChain Python SDK

You can use LangChain Python SDK to interact with FriendliAI. This makes migration of existing applications already using LangChain particularly easy.

​

How to use

Before you start, ensure you’ve already obtained the FRIENDLI_TOKEN from the Friendli Suite. Our products are entirely compatible with OpenAI, so we use the langchain-openai package by referring to the FriendliAI baseURL.

pip install -qU langchain-openai langchain

​

Instantiation

Now we can instantiate our model object and generate chat completions. We provide usage examples for each type of endpoint. Choose the one that best suits your needs:

​

Runnable interface

We support both synchronous and asynchronous runnable methods to generate a response.

​

Synchronous methods:

​

Asynchronous methods:

​

Chaining

We can chain our model with a prompt template. Prompt templates convert raw user input to better input to the LLM.

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a world class technical documentation writer."),
    ("user", "{input}")
])

chain = prompt | llm

print(chain.invoke({"input": "how can langsmith help with testing?"}))

To get the string value instead of the message, we can add an output parser to the chain.

from langchain_core.output_parsers import StrOutputParser

output_parser = StrOutputParser()

chain = prompt | llm | output_parser

print(chain.invoke({"input": "how can langsmith help with testing?"}))

​

Tool calling

Describe tools and their parameters, and let the model return a tool to invoke with the input arguments. Tool calling is extremely useful for enhancing the model’s capability to provide more comprehensive and actionable responses.

​

Define tools to use

The @tool decorator is used to define a tool. If you set parse_docstring=True, the tool will parse the docstring to extract the information of arguments.

​

Bind tools to the model

Now models can generate a tool calling response.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="meta-llama-3.1-8b-instruct",
    base_url="https://inference.friendli.ai/v1",
    api_key=os.environ["FRIENDLI_TOKEN"],
)

llm_with_tools = llm.bind_tools(tools)

query = "What is 3 * 12? Also, what is 11 + 49?"

print(llm_with_tools.invoke(query).tool_calls)

​

Generate a tool assisted message

Use the tool call results to generate a message.

from langchain_core.messages import HumanMessage, ToolMessage

messages = [HumanMessage(query)]
ai_msg = llm_with_tools.invoke(messages)
messages.append(ai_msg)

for tool_call in ai_msg.tool_calls:
    selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
    tool_output = selected_tool.invoke(tool_call["args"])
    messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))

print(llm_with_tools.invoke(messages))

For more information on how to use tools, check out the LangChain documentation.