Substitute conventional NLP approaches with immediate engineering and Massive Language Fashions (LLMS) for Jira ticket textual content classification. A code pattern walkthrough
Keep in mind the times when classifying textual content meant embarking on a machine studying journey? For those who’ve been within the ML area lengthy sufficient, you’ve most likely witnessed not less than one workforce disappear down the rabbit gap of constructing the “excellent” textual content classification system. The story often goes one thing like this:
- Month 1: “We’ll simply rapidly practice a NLP mannequin!”
- Month 2: “We’d like extra coaching knowledge…”
- Month 3: “That is ok”
For years, textual content classification has fallen into the realm of basic ML. Early in my profession, I keep in mind coaching a help vector machine (SVM) for e mail classification. A lot of preprocessing, iteration, knowledge assortment, and labeling.
However right here’s the twist: it’s 2024, and generative AI fashions can “usually” classify textual content out of the field! You possibly can construct a strong ticket classification system with out, accumulating hundreds of labeled coaching examples, managing ML coaching pipelines, or sustaining customized fashions.
On this submit, we’ll go over the way to setup a Jira ticket classification system utilizing massive language fashions on Amazon Bedrock and different AWS companies.
DISCLAIMER: I’m a GenAI Architect at AWS and my opinions are my very own.
Why Classify Jira Tickets?
A typical ask from corporations is to know how groups spend their time. Jira has tagging options, however it could possibly typically fall quick by human error or lack of granularity. By doing this train, organizations can get higher insights into their workforce actions, enabling data-driven selections about useful resource allocation, venture funding, and deprecation.
Why Not Use Different NLP Approaches?
Conventional ML fashions and smaller transformers like BERT want a whole bunch (or hundreds) of labeled examples, whereas LLMs can classify textual content out of the field. In our Jira ticket classification exams, a prompt-engineering method matched or beat conventional ML fashions, processing 10k+ annual tickets for ~$10/yr utilizing Claude Haiku (excluding different AWS Service prices). Additionally, prompts are simpler to replace than retraining fashions.
This github repo accommodates a pattern utility that connects to Jira Cloud, classifies tickets, and outputs them in a format that may be consumed by your favourite dashboarding instrument (Tableu, Quicksight, or every other instrument that helps CSVs).
Vital Discover: This venture deploys assets in your AWS atmosphere utilizing Terraform. You’ll incur prices for the AWS assets used. Please pay attention to the pricing for companies like Lambda, Bedrock, Glue, and S3 in your AWS area.
Pre Requisites
You’ll have to have terraform put in and the AWS CLI put in within the atmosphere you wish to deploy this code from
The structure is fairly straight ahead. You will discover particulars beneath.
Step 1: An AWS Lambda perform is triggered on a cron job to fetch jira tickets based mostly on a time window. These tickets are then formatted and pushed to an S3 bucket below the /unprocessed prefix.
Step 2: A Glue job is triggered off /unprocessed object places. This runs a PySpark deduplication process to make sure no duplicate tickets make their method to the dashboard. The deduplicated tickets are then put to the /staged prefix. That is helpful for instances the place you manually add tickets in addition to depend on the automated fetch. For those who can guarantee no duplicates, you’ll be able to take away this step.
Step 3: A classification process is kicked off on the brand new tickets by calling Amazon Bedrock to categorise the tickets based mostly on a immediate to a big language mannequin (LLM). After classification, the completed outcomes are pushed to the /processed prefix. From right here, you’ll be able to choose up the processed CSV utilizing any dashboarding instrument you’d like that may devour a CSV.
To get began, clone the github repo above and transfer to the /terraform listing
$ git clone https://github.com/aws-samples/jira-ticket-classification.git$ cd jira-ticket-classification/terraform
Run terraform init, plan, & apply. Ensure you have terraform put in in your pc and the AWS CLI configured.
$ terraform init$ terraform plan
$ terraform apply
As soon as the infrastructure is deployed into your account, you’ll be able to navigate to AWS Secrets and techniques Supervisor and replace the key together with your Jira Cloud credentials. You’ll want an API key, base url, and e mail to allow the automated pull
And that’s it!
You possibly can (1) watch for the Cron to kick off an automated fetch, (2) export the tickets to CSV and add them to the /unprocessed S3 bucket prefix, or (3) manually set off the Lambda perform utilizing a check.
Jira Fetch:
Jira fetch makes use of a Lambda perform with a Cloudwatch cron occasion to set off it. The Lambda pulls within the AWS Secret and makes use of a get request shortly loop to retrieve paginated outcomes till the JQL question completes:
def fetch_jira_issues(base_url, project_id, e mail, api_key):
url = f"{base_url}/relaxation/api/3/search"# Calculate the date 8 days in the past
eight_days_ago = (datetime.now() - timedelta(days=8)).strftime("%Y-%m-%d")
# Create JQL
jql = f"venture = {project_id} AND created >= '{eight_days_ago}' ORDER BY created DESC"
# Go into params of request.
params = {
"jql": jql,
"startAt": 0
}
all_issues = []
auth = HTTPBasicAuth(e mail, api_key)
headers = {"Settle for": "utility/json"}
whereas True:
response = requests.get(url, headers=headers, params=params, auth=auth)
if response.status_code != 200:
elevate Exception(f"Didn't fetch points for venture {project_id}: {response.textual content}")
knowledge = json.masses(response.textual content)
points = knowledge['issues']
all_issues.lengthen(points)
if len(all_issues) >= knowledge['total']:
break
params['startAt'] = len(all_issues)
return all_issues
It then creates a string illustration of a CSV and uploads it into S3:
def upload_to_s3(csv_string, bucket, key):
strive:
s3_client.put_object(
Bucket=bucket,
Key=key,
Physique=csv_string,
ContentType='textual content/csv'
)
besides Exception as e:
elevate Exception(f"Didn't add CSV to S3: {str(e)}")
Glue Job
An S3 occasion on the /unprocessed prefix kicks off a second lambda that begins an AWS Glue job. That is helpful when there’s a number of entry factors that Jira tickets can enter the system by. For instance, if you wish to do a backfill.
import boto3 # Initialize Boto3 Glue shopper
glue_client = boto3.shopper('glue')
def handler(occasion, context):
# Print occasion for debugging
print(f"Obtained occasion: {json.dumps(occasion)}")
# Get bucket identify and object key (file identify) from the S3 occasion
strive:
s3_event = occasion['Records'][0]['s3']
s3_bucket = s3_event['bucket']['name']
s3_key = s3_event['object']['key']
besides KeyError as e:
print(f"Error parsing S3 occasion: {str(e)}")
elevate
response = glue_client.start_job_run(
JobName=glue_job_name,
Arguments={
'--S3_BUCKET': s3_bucket,
'--NEW_CSV_FILE': s3_key
}
)
The Glue job itself is written in PySpark and might be discovered within the code repo right here. The essential take away is that it does a leftanti be part of utilizing the difficulty Ids on the objects within the new CSV towards all of the Ids within the /staged CSVs.
The outcomes are then pushed to the /staged prefix.
Classify Jira Tickets:
That is the place it it will get attention-grabbing. Because it seems, utilizing immediate engineering can carry out on par, if not higher, than a textual content classification mannequin utilizing a pair methods.
- You possibly can outline the classifications and their descriptions in a immediate,
- Ask the mannequin to suppose step-by-step (Chain of Thought).
- After which output the classification with out having to coach a single mannequin. See the immediate beneath:
Notice: It’s essential to validate your immediate utilizing a human curated subset of categorized / labelled tickets. It’s best to run this immediate by the validation dataset to ensure it aligns with the way you count on the tickets to be categorized
SYSTEM_PROMPT = '''
You're a help ticket assistant. You're given fields of a Jira ticket and your process is to categorise the ticket based mostly on these fieldsBeneath is the listing of potential classifications together with descriptions of these classifications.
ACCESS_PERMISSIONS_REQUEST: Used when somebody would not have the write permissions or cannot log in to one thing or they can not get the right IAM credentials to make a service work.
BUG_FIXING: Used when one thing is failing or a bug is discovered. Usually occasions the descriptions embrace logs or technical data.
CREATING_UPDATING_OR_DEPRECATING_DOCUMENTATION: Used when documentation is old-fashioned. Often references documentation within the textual content.
MINOR_REQUEST: That is not often used. Often a bug repair nevertheless it's very minor. If it appears even remotely sophisticated use BUG_FIXING.
SUPPORT_TROUBLESHOOTING: Used when asking for help for some engineering occasion. May also appear to be an automatic ticket.
NEW_FEATURE_WORK: Often describes a brand new function ask or one thing that is not operational.
The fields obtainable and their descriptions are beneath.
Summmary: It is a abstract or title of the ticket
Description: The outline of the difficulty in pure language. The vast majority of context wanted to categorise the textual content will come from this discipline
* It's doable that some fields could also be empty wherein case ignore them when classifying the ticket
* Assume by your reasoning earlier than making the classification and place your thought course of in
* Upon getting completed pondering, classify the ticket utilizing ONLY the classifications listed above and place it in
USER_PROMPT = '''
Utilizing solely the ticket fields beneath:
{abstract}
{description}
Classify the ticket utilizing ONLY 1 of the classifications listed within the system immediate. Keep in mind to suppose step-by-step earlier than classifying the ticket and place your ideas in
When you're completed pondering, classify the ticket and place your reply in
'''
We’ve added a helper class that threads the calls to Bedrock to hurry issues up:
import boto3
from concurrent.futures import ThreadPoolExecutor, as_completed
import re
from typing import Checklist, Dict
from prompts import USER_PROMPT, SYSTEM_PROMPTclass TicketClassifier:
SONNET_ID = "anthropic.claude-3-sonnet-20240229-v1:0"
HAIKU_ID = "anthropic.claude-3-haiku-20240307-v1:0"
HYPER_PARAMS = {"temperature": 0.35, "topP": .3}
REASONING_PATTERN = r'(.*?) '
CORRECTNESS_PATTERN = r'(.*?) '
def __init__(self):
self.bedrock = boto3.shopper('bedrock-runtime')
def classify_tickets(self, tickets: Checklist[Dict[str, str]]) -> Checklist[Dict[str, str]]:
prompts = [self._create_chat_payload(t) for t in tickets]
responses = self._call_threaded(prompts, self._call_bedrock)
formatted_responses = [self._format_results(r) for r in responses]
return [{**d1, **d2} for d1, d2 in zip(tickets, formatted_responses)]
def _call_bedrock(self, message_list: listing[dict]) -> str:
response = self.bedrock.converse(
modelId=self.HAIKU_ID,
messages=message_list,
inferenceConfig=self.HYPER_PARAMS,
system=[{"text": SYSTEM_PROMPT}]
)
return response['output']['message']['content'][0]['text']
def _call_threaded(self, requests, perform):
future_to_position = {}
with ThreadPoolExecutor(max_workers=5) as executor:
for i, request in enumerate(requests):
future = executor.submit(perform, request)
future_to_position[future] = i
responses = [None] * len(requests)
for future in as_completed(future_to_position):
place = future_to_position[future]
strive:
response = future.outcome()
responses[position] = response
besides Exception as exc:
print(f"Request at place {place} generated an exception: {exc}")
responses[position] = None
return responses
def _create_chat_payload(self, ticket: dict) -> dict:
user_prompt = USER_PROMPT.format(abstract=ticket['Summary'], description=ticket['Description'])
user_msg = {"function": "person", "content material": [{"text": user_prompt}]}
return [user_msg]
def _format_results(self, model_response: str) -> dict:
reasoning = self._extract_with_regex(model_response, self.REASONING_PATTERN)
correctness = self._extract_with_regex(model_response, self.CORRECTNESS_PATTERN)
return {'Mannequin Reply': correctness, 'Reasoning': reasoning}
@staticmethod
def _extract_with_regex(response, regex):
matches = re.search(regex, response, re.DOTALL)
return matches.group(1).strip() if matches else None
Lastly, the categorized tickets are transformed to a CSV and uploaded to S3
import boto3
import io
import csvs3 = boto3.shopper('s3')
def upload_csv(knowledge: Checklist[Dict[str, str]]) -> None:
csv_buffer = io.StringIO()
author = csv.DictWriter(csv_buffer, fieldnames=knowledge[0].keys())
author.writeheader()
author.writerows(knowledge)
current_time = datetime.now().strftime("%Ypercentmpercentd_percentHpercentMpercentS")
filename = f"processed/processed_{current_time}.csv"
s3.put_object(
Bucket=self.bucket_name,
Key=filename,
Physique=csv_buffer.getvalue()
)
The venture is dashboard agnostic. Any well-liked instrument/service will work so long as it could possibly devour a CSV. Amazon Quicksight, Tableu or something in between will do.
On this weblog we mentioned utilizing Bedrock to mechanically classify Jira tickets. These enriched tickets can then be used to create dashboards utilizing varied AWS Providers or 3P instruments. The takeaway, is that classifying textual content has grow to be a lot easier for the reason that adoption of LLMs and what would have taken weeks can now be accomplished in days.
For those who loved this text be at liberty to attach with me on LinkedIn
