How to troubleshoot intermittent lockups

Hi,

I'm an accidental DBA and I was wondering how to troubleshoot what causes Sql Server to lock up some times. When I say “lock up”, I don’t mean completely. I’m talking about a situation everything’s working fine and all of a sudden we start getting complaints that users can’t get into or use SuchAndSuch application which uses Sql Server. So it's not to the point where I have to use the dedicated administratrive connection...but it is simply not responding to requests!

I check the replication monitor and all of our (push subscription) replication processes are backed up and all of them say “Synchronizing” and they’re not making any progress.

I check the activity monitor and if I trace the “blocked by” column I can usually see that there are several processes blocking other things. But it’s hard to determine the root cause, because once one thing blocks another thing, that causes a chain reaction (especially once the replication processes start kicking in).

For instance, this happened earlier today and there were several processes that said “blocked by” process 13. Yet there was no process 13 in the activity monitor (yes I had it ordered by ProcessID and I checked it twice).

I know how to start a profiler trace, but I’m never sure how to interpret the plethora of results to make any type of intelligent determination from them.

Usually what happens is I sit there browsing through the task manager, replication monitor and activity monitor until it *magically* starts responding on its own just as suddenly as it stopped! Of course, the users ask “how did you fix it”? –To which I have no good answer.

If your database is not responding what is the easiest way to determine the root cause? Trial and error takes too long with an intermittent problem and this solution never satisfies upper management!

Thanks,
Nate

Hi Nate,

You should invest a few hours of your time in this webcasts (the return of investment is great IMHO):

"TechNet Support WebCast:Performance troubleshooting and analysis in Microsoft SQL Server 2000" http://support.microsoft.com/kb/838622/en-us

"TechNet Webcast: Troubleshooting Performance Problems in Microsoft SQL Server 2005" http://msevents.microsoft.com/cui/eventdetail.aspx?eventID=1032275646&Culture=en-US

"TechNet Webcast: SQL Server 2005 Troubleshooting: Supportability Features for SQL 2005"
http://msevents.microsoft.com/cui/eventdetail.aspx?eventID=1032275593&Culture=en-US

"TechNet Webcast: Performance Diagnosis in SQL Server 2005"
http://msevents.microsoft.com/cui/eventdetail.aspx?eventID=1032275586&Culture=en-US

For SQL Server 2005 you should take a look at this whitepaper: "Troubleshooting Performance Problems in SQL Server 2005" http://www.microsoft.com/technet/prodtechnol/sql/2005/tsprfprb.mspx

|||Thanks Cristian. I will definitely check these out!sql

How to troubleshoot a strange result from a stored procedure?

I am trying to tune a stored procedure that is long and running slow. As
part of testing, the results are odd from what I am expecting. The testing
is running on 2 databases which restored from the same backup with one
database set with read-only option. This will allows me to check the
original code with modified code. These databases are on my server;
therefore, it is lock down.
The stored procedure has 15 pass-in variables with default values are NULL.
If I modify the stored procedure by adding a non-essential statement such as
SELECT 1 or SELECT getdate() anywhere in the stored procedure, the procedure
return different result.
Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price =
default, @.CustType = default
Part of the existing stored procedure:
if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
@.CustType <> '')
select * from CustTable
where @.LastName is null or customer.FName like rtrim(@.LastName)
else
select 'No data'
When I run this on the read-only database, it returns some rows. But when I
run this on the test database with modified code, it returns no data. The
stored procedures from both database are basically same with exception of
adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
what test query, it should return no data on both databases, but the results
are different.
Is there a tool or methods to figure out the strange results from this
stored procedure? I tested with different methods but come out empty-handed
.
Please help! Thanks!Correction to part of the stored procedure:
if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
@.CustType <> '')
select * from CustTable
where @.LastName is null or LName like rtrim(@.LastName)
else
select 'No data'
"KTN" wrote:

> I am trying to tune a stored procedure that is long and running slow. As
> part of testing, the results are odd from what I am expecting. The testin
g
> is running on 2 databases which restored from the same backup with one
> database set with read-only option. This will allows me to check the
> original code with modified code. These databases are on my server;
> therefore, it is lock down.
> The stored procedure has 15 pass-in variables with default values are NULL
.
> If I modify the stored procedure by adding a non-essential statement such
as
> SELECT 1 or SELECT getdate() anywhere in the stored procedure, the procedu
re
> return different result.
>
> Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price =
> default, @.CustType = default
> Part of the existing stored procedure:
> if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> @.CustType <> '')
> select * from CustTable
> where @.LastName is null or customer.FName like rtrim(@.LastName)
> else
> select 'No data'
>
> When I run this on the read-only database, it returns some rows. But when
I
> run this on the test database with modified code, it returns no data. The
> stored procedures from both database are basically same with exception of
> adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> what test query, it should return no data on both databases, but the resul
ts
> are different.
> Is there a tool or methods to figure out the strange results from this
> stored procedure? I tested with different methods but come out empty-hand
ed.
> Please help! Thanks!|||KTN
Do you investigate how to optimize the SP or why it returns wrong result?
Have you look into an execution plan? Was the optimizer available to use
indexes? How much data do you return?
"KTN" <KTN@.discussions.microsoft.com> wrote in message
news:47DD3664-5829-40E5-92B8-A5EE1E8D9C45@.microsoft.com...
>I am trying to tune a stored procedure that is long and running slow. As
> part of testing, the results are odd from what I am expecting. The
> testing
> is running on 2 databases which restored from the same backup with one
> database set with read-only option. This will allows me to check the
> original code with modified code. These databases are on my server;
> therefore, it is lock down.
> The stored procedure has 15 pass-in variables with default values are
> NULL.
> If I modify the stored procedure by adding a non-essential statement such
> as
> SELECT 1 or SELECT getdate() anywhere in the stored procedure, the
> procedure
> return different result.
>
> Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price =
> default, @.CustType = default
> Part of the existing stored procedure:
> if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> @.CustType <> '')
> select * from CustTable
> where @.LastName is null or customer.FName like rtrim(@.LastName)
> else
> select 'No data'
>
> When I run this on the read-only database, it returns some rows. But when
> I
> run this on the test database with modified code, it returns no data. The
> stored procedures from both database are basically same with exception of
> adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> what test query, it should return no data on both databases, but the
> results
> are different.
> Is there a tool or methods to figure out the strange results from this
> stored procedure? I tested with different methods but come out
> empty-handed.
> Please help! Thanks!|||My plan is to optimize the SP and in part of doing that, I found wrong
results with adding non-essential statements such as SELECT getddate()
I looked at the execution plan for the new code is much better via using
some covering index and better query plans.
The data return is very based on the 15 pass-in variables, but mostly under
100 rows.
"Uri Dimant" wrote:

> KTN
> Do you investigate how to optimize the SP or why it returns wrong result?
> Have you look into an execution plan? Was the optimizer available to use
> indexes? How much data do you return?
>
>
>
> "KTN" <KTN@.discussions.microsoft.com> wrote in message
> news:47DD3664-5829-40E5-92B8-A5EE1E8D9C45@.microsoft.com...
>
>

How to troubleshoot a strange result from a stored procedure?

I am trying to tune a stored procedure that is long and running slow. As
part of testing, the results are odd from what I am expecting. The testing
is running on 2 databases which restored from the same backup with one
database set with read-only option. This will allows me to check the
original code with modified code. These databases are on my server;
therefore, it is lock down.
The stored procedure has 15 pass-in variables with default values are NULL.
If I modify the stored procedure by adding a non-essential statement such as
SELECT 1 or SELECT getdate() anywhere in the stored procedure, the procedure
return different result.
Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price =
default, @.CustType = default
Part of the existing stored procedure:
if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
@.CustType <> '')
select * from CustTable
where @.LastName is null or customer.FName like rtrim(@.LastName)
else
select 'No data'
When I run this on the read-only database, it returns some rows. But when I
run this on the test database with modified code, it returns no data. The
stored procedures from both database are basically same with exception of
adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
what test query, it should return no data on both databases, but the results
are different.
Is there a tool or methods to figure out the strange results from this
stored procedure? I tested with different methods but come out empty-handed.
Please help! Thanks!
Correction to part of the stored procedure:
if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
@.CustType <> '')
select * from CustTable
where @.LastName is null or LName like rtrim(@.LastName)
else
select 'No data'
"KTN" wrote:

> I am trying to tune a stored procedure that is long and running slow. As
> part of testing, the results are odd from what I am expecting. The testing
> is running on 2 databases which restored from the same backup with one
> database set with read-only option. This will allows me to check the
> original code with modified code. These databases are on my server;
> therefore, it is lock down.
> The stored procedure has 15 pass-in variables with default values are NULL.
> If I modify the stored procedure by adding a non-essential statement such as
> SELECT 1 or SELECT getdate() anywhere in the stored procedure, the procedure
> return different result.
>
> Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price =
> default, @.CustType = default
> Part of the existing stored procedure:
> if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> @.CustType <> '')
> select * from CustTable
> where @.LastName is null or customer.FName like rtrim(@.LastName)
> else
> select 'No data'
>
> When I run this on the read-only database, it returns some rows. But when I
> run this on the test database with modified code, it returns no data. The
> stored procedures from both database are basically same with exception of
> adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> what test query, it should return no data on both databases, but the results
> are different.
> Is there a tool or methods to figure out the strange results from this
> stored procedure? I tested with different methods but come out empty-handed.
> Please help! Thanks!
|||KTN
Do you investigate how to optimize the SP or why it returns wrong result?
Have you look into an execution plan? Was the optimizer available to use
indexes? How much data do you return?
"KTN" <KTN@.discussions.microsoft.com> wrote in message
news:47DD3664-5829-40E5-92B8-A5EE1E8D9C45@.microsoft.com...
>I am trying to tune a stored procedure that is long and running slow. As
> part of testing, the results are odd from what I am expecting. The
> testing
> is running on 2 databases which restored from the same backup with one
> database set with read-only option. This will allows me to check the
> original code with modified code. These databases are on my server;
> therefore, it is lock down.
> The stored procedure has 15 pass-in variables with default values are
> NULL.
> If I modify the stored procedure by adding a non-essential statement such
> as
> SELECT 1 or SELECT getdate() anywhere in the stored procedure, the
> procedure
> return different result.
>
> Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price =
> default, @.CustType = default
> Part of the existing stored procedure:
> if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> @.CustType <> '')
> select * from CustTable
> where @.LastName is null or customer.FName like rtrim(@.LastName)
> else
> select 'No data'
>
> When I run this on the read-only database, it returns some rows. But when
> I
> run this on the test database with modified code, it returns no data. The
> stored procedures from both database are basically same with exception of
> adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> what test query, it should return no data on both databases, but the
> results
> are different.
> Is there a tool or methods to figure out the strange results from this
> stored procedure? I tested with different methods but come out
> empty-handed.
> Please help! Thanks!
|||My plan is to optimize the SP and in part of doing that, I found wrong
results with adding non-essential statements such as SELECT getddate()
I looked at the execution plan for the new code is much better via using
some covering index and better query plans.
The data return is very based on the 15 pass-in variables, but mostly under
100 rows.
"Uri Dimant" wrote:

> KTN
> Do you investigate how to optimize the SP or why it returns wrong result?
> Have you look into an execution plan? Was the optimizer available to use
> indexes? How much data do you return?
>
>
>
> "KTN" <KTN@.discussions.microsoft.com> wrote in message
> news:47DD3664-5829-40E5-92B8-A5EE1E8D9C45@.microsoft.com...
>
>

How to troubleshoot a strange result from a stored procedure?

I am trying to tune a stored procedure that is long and running slow. As
part of testing, the results are odd from what I am expecting. The testing
is running on 2 databases which restored from the same backup with one
database set with read-only option. This will allows me to check the
original code with modified code. These databases are on my server;
therefore, it is lock down.
The stored procedure has 15 pass-in variables with default values are NULL.
If I modify the stored procedure by adding a non-essential statement such as
SELECT 1 or SELECT getdate() anywhere in the stored procedure, the procedure
return different result.
Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price = default, @.CustType = default
Part of the existing stored procedure:
if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
@.CustType <> '')
select * from CustTable
where @.LastName is null or customer.FName like rtrim(@.LastName)
else
select 'No data'
When I run this on the read-only database, it returns some rows. But when I
run this on the test database with modified code, it returns no data. The
stored procedures from both database are basically same with exception of
adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
what test query, it should return no data on both databases, but the results
are different.
Is there a tool or methods to figure out the strange results from this
stored procedure? I tested with different methods but come out empty-handed.
Please help! Thanks!Correction to part of the stored procedure:
if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
@.CustType <> '')
select * from CustTable
where @.LastName is null or LName like rtrim(@.LastName)
else
select 'No data'
"KTN" wrote:
> I am trying to tune a stored procedure that is long and running slow. As
> part of testing, the results are odd from what I am expecting. The testing
> is running on 2 databases which restored from the same backup with one
> database set with read-only option. This will allows me to check the
> original code with modified code. These databases are on my server;
> therefore, it is lock down.
> The stored procedure has 15 pass-in variables with default values are NULL.
> If I modify the stored procedure by adding a non-essential statement such as
> SELECT 1 or SELECT getdate() anywhere in the stored procedure, the procedure
> return different result.
>
> Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price => default, @.CustType = default
> Part of the existing stored procedure:
> if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> @.CustType <> '')
> select * from CustTable
> where @.LastName is null or customer.FName like rtrim(@.LastName)
> else
> select 'No data'
>
> When I run this on the read-only database, it returns some rows. But when I
> run this on the test database with modified code, it returns no data. The
> stored procedures from both database are basically same with exception of
> adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> what test query, it should return no data on both databases, but the results
> are different.
> Is there a tool or methods to figure out the strange results from this
> stored procedure? I tested with different methods but come out empty-handed.
> Please help! Thanks!|||KTN
Do you investigate how to optimize the SP or why it returns wrong result?
Have you look into an execution plan? Was the optimizer available to use
indexes? How much data do you return?
"KTN" <KTN@.discussions.microsoft.com> wrote in message
news:47DD3664-5829-40E5-92B8-A5EE1E8D9C45@.microsoft.com...
>I am trying to tune a stored procedure that is long and running slow. As
> part of testing, the results are odd from what I am expecting. The
> testing
> is running on 2 databases which restored from the same backup with one
> database set with read-only option. This will allows me to check the
> original code with modified code. These databases are on my server;
> therefore, it is lock down.
> The stored procedure has 15 pass-in variables with default values are
> NULL.
> If I modify the stored procedure by adding a non-essential statement such
> as
> SELECT 1 or SELECT getdate() anywhere in the stored procedure, the
> procedure
> return different result.
>
> Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price => default, @.CustType = default
> Part of the existing stored procedure:
> if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> @.CustType <> '')
> select * from CustTable
> where @.LastName is null or customer.FName like rtrim(@.LastName)
> else
> select 'No data'
>
> When I run this on the read-only database, it returns some rows. But when
> I
> run this on the test database with modified code, it returns no data. The
> stored procedures from both database are basically same with exception of
> adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> what test query, it should return no data on both databases, but the
> results
> are different.
> Is there a tool or methods to figure out the strange results from this
> stored procedure? I tested with different methods but come out
> empty-handed.
> Please help! Thanks!|||My plan is to optimize the SP and in part of doing that, I found wrong
results with adding non-essential statements such as SELECT getddate()
I looked at the execution plan for the new code is much better via using
some covering index and better query plans.
The data return is very based on the 15 pass-in variables, but mostly under
100 rows.
"Uri Dimant" wrote:
> KTN
> Do you investigate how to optimize the SP or why it returns wrong result?
> Have you look into an execution plan? Was the optimizer available to use
> indexes? How much data do you return?
>
>
>
> "KTN" <KTN@.discussions.microsoft.com> wrote in message
> news:47DD3664-5829-40E5-92B8-A5EE1E8D9C45@.microsoft.com...
> >I am trying to tune a stored procedure that is long and running slow. As
> > part of testing, the results are odd from what I am expecting. The
> > testing
> > is running on 2 databases which restored from the same backup with one
> > database set with read-only option. This will allows me to check the
> > original code with modified code. These databases are on my server;
> > therefore, it is lock down.
> >
> > The stored procedure has 15 pass-in variables with default values are
> > NULL.
> > If I modify the stored procedure by adding a non-essential statement such
> > as
> > SELECT 1 or SELECT getdate() anywhere in the stored procedure, the
> > procedure
> > return different result.
> >
> >
> > Test query: exec GetValues @.FName = 'Smith%', @.FName = default, @.Price => > default, @.CustType = default
> >
> > Part of the existing stored procedure:
> > if (@.Price is not null or @.Price <> '') and (@.CustomerType is not null or
> > @.CustType <> '')
> > select * from CustTable
> > where @.LastName is null or customer.FName like rtrim(@.LastName)
> > else
> > select 'No data'
> >
> >
> > When I run this on the read-only database, it returns some rows. But when
> > I
> > run this on the test database with modified code, it returns no data. The
> > stored procedures from both database are basically same with exception of
> > adding a non-essential statement (SELECT 1 or SELECT getdate()). Based on
> > what test query, it should return no data on both databases, but the
> > results
> > are different.
> >
> > Is there a tool or methods to figure out the strange results from this
> > stored procedure? I tested with different methods but come out
> > empty-handed.
> > Please help! Thanks!
>
>

How to trouble shoot a deadlock problem

I have a job that took 1 hour and 55 mintes to run, which is causing deadlock situatin with another database. How would I go about troubleshoot this problem.
Thanks
LystrWell, you could run Enterprise Manager, go to Manage->Current activity and look at process info to find the locking process. Look at the locks / process id / object.

Try running your queries with the (nolock) hint or use temp tables.|||The message I am getting is:

The VB Application identified by the event source logged this Application ANSDAC: Thread ID: 2768, Logged: Microsoft OLE DB Provider for ODBC Drivers Session.Save() -2147467259 [Microsoft][ODBC SQL Server Driver][SQL Server]Transaction (Process ID 66) was deadlocked on lock resources with another process and has been chosed as the deadlock victim. Rerun the transaction.

Would I need to rerun the process ID 66?|||Yes, whatever query/statement being run by that process was aborted due to deadlocking. It must be re-run.

Try running your job on non-peak/business hours, if possible. What job are you running, btw?|||Then name of the job is call Archive IIS Log which consists of 4 steps
1-Update department-run sp_Update_Log
2-Copy to Archive Database-run sp_ArchiveLog
3-Clean up Log-run sp_cleaningLog
4-Free log space-Backup Log IISLog with No_Log

It runs at 9am everyday which is a busy time. The developer feel that the solution should be to move the database to another server. The Job did finish it just took 1 and 55 minutes, but no one can tell me how long do the job normally take.

So I should run each stored procedure.

Also, when I look at process ID 66
this is what I see"

SELECT MonthEnd."Accounting_date", Manual_Adjustment."Accounting date", Manual_Adjustment."Adjustment_amount", Manual_Adjustment."CheckNumber" FROM { oj "MonthEnd" MonthEnd INNER JOIN "ManualAdjustment" Manual_Adjustment ON MonthEnd."Accounting_date" = Ma

However this is not the database that they are complaining about. Is it possible for 66 to be connected with more than 1 database?

Thanks

Lystra|||use Trace Flags. Check out in Books Online about them else search on google key words [TRACE FLAG, DEADLOCK].|||Yes, process 66 can change the database it is running against at any time.

-When it is blocking, look at the blocking process (the job process) and see what database and query/statement it is running.

-Also look at running it at off peak hours if at all possible. If it is something that updates a bunch of records in a single table you are bound to cause locking. Espcially if this thing runs that long.

-On the database they are complaining about - do the job scripts have a lot of references to this table or are there only a few? If there are a few it should be easier to pinpoint which statement is causing the locking.

-Try starting Sql Profiler before the job runs and let it run to track all sql statements made by the job. As soon as the blocking stops, look to see if any long queries just finished running in the profiler. These could be the culprits.

Once you know the statements causing the issue you can better determine how to resolve the issue.|||First order of business, go beat the developer soundly for scheduling an automated process during a high usage time for users. That may take a while to beat them soundly, but it is an important step, so I'll wait while you finish.

Now that they've been beaten, go back and beat them again. That might make them think twice about such idiocy in the future.

Ok, that is as good a preventative as we can manage at the moment, so now we should move on to correcting the underlying problem. Pick the lowest usage period of the day when you can be certain that all the affected systems will be available. 03:00 works well for me, although some folks prefer 20:00 or something like it. Change the job so that it now runs at a low usage time instead of a high usage time.

After the job runs at its new time, check the SQL Server and NT logs for any error messages. Sometimes there are hidden dependancies that can trip you up when you move a job. If those both check out, then check the log tables that this process maintains to be sure that they also look as you'd expect.

Once you get done cleaning up the problem, go hose the blood off of the developer and carry them to a nice quiet closet somewhere to sleep off the beatings... There's no point in being mean about things, you just have to curb that behavior problem!

-PatP|||The database is called ISSLOG, which log all of the traffic from the web. Now why would this logging database cause deadlocking? The database log information like, clienthost, username, logtime, service, machine, serverIP, processing time, etc..,

I have added the code please look to see why it would cause a deadlock issue.

Thanks

Lystra|||I think my eyes just popped. I think if code1, code2 and code3 are run one after the other, there might be some other processing running that's causing the deadlock. If Code2 and Code3 are run at the same time, it might cause some trouble.|||Then name of the job is call Archive IIS Log which consists of 4 steps
1-Update department-run sp_Update_Log (CODE 3)
2-Copy to Archive Database-run sp_ArchiveLog (CODE 1)
3-Clean up Log-run sp_cleaningLog (CODE 3)
4-Free log space-Backup Log IISLog with No_Log

The code is ran one after the other.

If the code is looking to log traffic why should it create deadlock, because it's not looking at rows of data.

Thanks|||First order of business, go beat the developer soundly for scheduling an automated process during a high usage time for users. That may take a while to beat them soundly, but it is an important step, so I'll wait while you finish.

Now that they've been beaten, go back and beat them again. That might make them think twice about such idiocy in the future.

Ok, that is as good a preventative as we can manage at the moment, so now we should move on to correcting the underlying problem. Pick the lowest usage period of the day when you can be certain that all the affected systems will be available. 03:00 works well for me, although some folks prefer 20:00 or something like it. Change the job so that it now runs at a low usage time instead of a high usage time.

After the job runs at its new time, check the SQL Server and NT logs for any error messages. Sometimes there are hidden dependancies that can trip you up when you move a job. If those both check out, then check the log tables that this process maintains to be sure that they also look as you'd expect.

Once you get done cleaning up the problem, go hose the blood off of the developer and carry them to a nice quiet closet somewhere to sleep off the beatings... There's no point in being mean about things, you just have to curb that behavior problem!

-PatP

What pat said, but just for good measure, beat them again after the problem's sorted, then hose them down. ;)